Starting the Airbyte API on a self-hosted instance

cornjuliox · September 23, 2022, 9:20am

I need to be able to change the state of a particular connector, I’m aware you can do that through the Airbyte API as described here

But when I try to do it on our self-hosted instance (via postman) the API request just returns a 405 Not Allowed header. I assumed that the and when I click through the web interface I don’t see anything related to the API anywhere.

Does it even start when you run docker-compose up? How do we enable / access the API?

marcosmarxm · September 23, 2022, 3:02pm

Hello there! You are receiving this message because none of your fellow community members has stepped in to respond to your topic post. (If you are a community member and you are reading this response, feel free to jump in if you have the answer!) As a result, the Community Assistance Team has been made aware of this topic and will be investigating and responding as quickly as possible.
Some important considerations that will help your to get your issue solved faster:

It is best to use our topic creation template; if you haven’t yet, we recommend posting a followup with the requested information. With that information the team will be able to more quickly search for similar issues with connectors and the platform and troubleshoot more quickly your specific question or problem.
Make sure to upload the complete log file; a common investigation roadblock is that sometimes the error for the issue happens well before the problem is surfaced to the user, and so having the tail of the log is less useful than having the whole log to scan through.
Be as descriptive and specific as possible; when investigating it is extremely valuable to know what steps were taken to encounter the issue, what version of connector / platform / Java / Python / docker / k8s was used, etc. The more context supplied, the quicker the investigation can start on your topic and the faster we can drive towards an answer.
We in the Community Assistance Team are glad you’ve made yourself part of our community, and we’ll do our best to answer your questions and resolve the problems as quickly as possible. Expect to hear from a specific team member as soon as possible.

Thank you for your time and attention.
Best,
The Community Assistance Team

marcosmarxm · September 23, 2022, 3:09pm

You must change the headers to the correct type. Make sure you have Content-Type and Accept json as headers.

cornjuliox · September 26, 2022, 1:10pm

Hey that worked! Thanks. I’ve made some progress but I’m having another issue now.

The connection ID I’m using works just fine when passed to /api/v1/state/get , but I get a 404 Object ID Not Found error when trying to update the same state through /api/v1/state/create_or_update/

I noticed that the body you’re using is different - this one is mine

{
  "connectionId": "6bcff6d1-8703-4e5c-85f5-0dbe88035ca2",
  "connectionState": {
    "stateType": "legacy",
    "connectionId": "6bcff6d1-8703-4e5c-85f5-0dbe88035ca2",
    "state": {},
    "streamState": [
      {
        "streamDescriptor": {
          "name": "events",
          "namespace": "events"
        },
        "streamState": {
            "updated_at": "2022-09-17 04:34:40"
        }
      }
    ]
  }
}

I’m not sure if I did this right, though. What’s the difference between state and streamState?

marcosmarxm · September 26, 2022, 6:48pm

state is the legacy where all states were saved together.
streamState is the new method that will be enable to save state for each stream, allowing to reset/refresh only that one given more granularity.

cornjuliox · September 27, 2022, 9:48am

Thanks!

I’m still having issues, unfortunately - here’s what I’ve got.

POST request to /api/v1/state/get gives me a connection ID and the bad state I’m trying to update

{
    "stateType": "legacy",
    "connectionId": "6bcff6d1-8703-4e5c-85f5-0dbe88035ca2",
    "state": {
        "events": {
            "updated_at": {
                "updated_at": "2022-09-17 04:34:40"
            }
        }
    }
}

Passing that connection ID in a POST to /api/v1/state/create_or_update like so

{
  "connectionId": "6bcff6d1-8703-4e5c-85f5-0dbe88035ca2",
  "connectionState": {
    "stateType": "legacy",
    "connectionId": "6bcff6d1-8703-4e5c-85f5-0dbe88035ca2",
    "state": {
        "events": {
            "updated_at": "2022-09-17 04:34:40" 
        }
    }
  }
}

Gets me this 404 error:

{
    "message": "Object not found. HTTP 404 Not Found",
    "exceptionClassName": "io.airbyte.server.errors.IdNotFoundKnownException",
    "exceptionStack": [
        "io.airbyte.server.errors.IdNotFoundKnownException: Object not found. HTTP 404 Not Found",
        "\tat io.airbyte.server.errors.NotFoundExceptionMapper.toResponse(NotFoundExceptionMapper.java:22)",
        "\tat io.airbyte.server.errors.NotFoundExceptionMapper.toResponse(NotFoundExceptionMapper.java:14)",
        "\tat org.glassfish.jersey.server.ServerRuntime$Responder.mapException(ServerRuntime.java:528)",
        "\tat org.glassfish.jersey.server.ServerRuntime$Responder.process(ServerRuntime.java:403)",
        "\tat org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:261)",
        "\tat org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)",
        "\tat org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)",
        "\tat org.glassfish.jersey.internal.Errors.process(Errors.java:292)",
        "\tat org.glassfish.jersey.internal.Errors.process(Errors.java:274)",
        "\tat org.glassfish.jersey.internal.Errors.process(Errors.java:244)",
        "\tat org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265)",
        "\tat org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:232)",
        "\tat org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:680)",
        "\tat org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:394)",
        "\tat org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:346)",
        "\tat org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:366)",
        "\tat org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:319)",
        "\tat org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205)",
        "\tat org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:763)",
        "\tat org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:569)",
        "\tat org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)",
        "\tat org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1377)",
        "\tat org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)",
        "\tat org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:507)",
        "\tat org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)",
        "\tat org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1292)",
        "\tat org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)",
        "\tat org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)",
        "\tat org.eclipse.jetty.server.Server.handle(Server.java:501)",
        "\tat org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383)",
        "\tat org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:556)",
        "\tat org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375)",
        "\tat org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:273)",
        "\tat org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)",
        "\tat org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)",
        "\tat org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)",
        "\tat org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)",
        "\tat org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)",
        "\tat org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)",
        "\tat org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)",
        "\tat org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:375)",
        "\tat org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)",
        "\tat org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)",
        "\tat java.base/java.lang.Thread.run(Thread.java:1589)",
        "Caused by: javax.ws.rs.NotFoundException: HTTP 404 Not Found",
        "\tat org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:250)",
        "\t... 39 more"
    ],
    "rootCauseExceptionClassName": "java.lang.Class",
    "rootCauseExceptionStack": [
        "javax.ws.rs.NotFoundException: HTTP 404 Not Found",
        "\tat org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:250)",
        "\tat org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)",
        "\tat org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)",
        "\tat org.glassfish.jersey.internal.Errors.process(Errors.java:292)",
        "\tat org.glassfish.jersey.internal.Errors.process(Errors.java:274)",
        "\tat org.glassfish.jersey.internal.Errors.process(Errors.java:244)",
        "\tat org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265)",
        "\tat org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:232)",
        "\tat org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:680)",
        "\tat org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:394)",
        "\tat org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:346)",
        "\tat org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:366)",
        "\tat org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:319)",
        "\tat org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205)",
        "\tat org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:763)",
        "\tat org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:569)",
        "\tat org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)",
        "\tat org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1377)",
        "\tat org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)",
        "\tat org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:507)",
        "\tat org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)",
        "\tat org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1292)",
        "\tat org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)",
        "\tat org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)",
        "\tat org.eclipse.jetty.server.Server.handle(Server.java:501)",
        "\tat org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383)",
        "\tat org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:556)",
        "\tat org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375)",
        "\tat org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:273)",
        "\tat org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)",
        "\tat org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)",
        "\tat org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)",
        "\tat org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)",
        "\tat org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)",
        "\tat org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)",
        "\tat org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)",
        "\tat org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:375)",
        "\tat org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)",
        "\tat org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)",
        "\tat java.base/java.lang.Thread.run(Thread.java:1589)"
    ]
}

What am I doing wrong here? I took the connection ID from the web frontend’s URL, and that seemed to work for the initial call to /get at least.

marcosmarxm · September 27, 2022, 5:04pm

I’m using Airbyte 0.40.6+ version I strongly recommend to you use the same version or latest.

I have this connection in my local setup:

First I triggered the /state/get

curl -X POST "http://localhost:8000/api/v1/state/get" \ -H "Accept: application/json" \ -H "Content-Type: application/json" \ -d '{"connectionId":"01046826-c1b0-4b0e-815e-7b26f9319ebb"}' \

output is:

{ "stateType": "legacy", "connectionId": "01046826-c1b0-4b0e-815e-7b26f9319ebb", "state": { "deals": { "updatedAt": "2020-09-27T16:55:13.217559Z" } }}

Then, I tried the change state one:

Payload

{ "connectionId": "01046826-c1b0-4b0e-815e-7b26f9319ebb", "connectionState": { "stateType": "legacy", "connectionId": "01046826-c1b0-4b0e-815e-7b26f9319ebb", "state": { "deals": { "updatedAt": "2022-01-01T16:55:13.217559Z" } } }}

Complete call request:

curl -X POST "http://localhost:8000/api/v1/state/create_or_update" \ -H "Accept: application/json" \ -H "Content-Type: application/json" \ -d '{"connectionId":"01046826-c1b0-4b0e-815e-7b26f9319ebb","connectionState":{"stateType":"legacy","connectionId":"01046826-c1b0-4b0e-815e-7b26f9319ebb","state":{"deals":{"updatedAt":"2020-09-27T16:55:13.217559Z"}}}}' \

Output is:

{ "stateType": "legacy", "connectionId": "01046826-c1b0-4b0e-815e-7b26f9319ebb", "state": { "deals": { "updatedAt": "2022-01-01T16:55:13.217559Z" } }}

If I tried the get state I see the updated one.

cornjuliox · October 17, 2022, 4:27pm

Alright, I managed to get this working, thanks for the help @marcosmarxm !

Topic		Replies	Views
Private Custom Connector on Airbyte Cloud Connector Development	2	366	December 6, 2022
Airbyte-http-api-connector Connector Development connectors	3	522	July 14, 2022
Contributing a Connector implementation from out of Airbyte Connector Development connector-development	7	524	October 5, 2022
Handling "async" APIs? Connector Development	3	553	July 4, 2023
Correctly handling private key file as a secret Connector Development	4	294	September 27, 2022

Starting the Airbyte API on a self-hosted instance

Related Topics