Step 4: Reading data
Now that we're able to authenticate to the source API, we'll want to select data from the HTTP responses.
Let's first add the stream to the configured catalog in source-exchange-rates-tutorial/integration_tests/configured_catalog.json
{
"streams": [
{
"stream": {
"name": "rates",
"json_schema": {},
"supported_sync_modes": [
"full_refresh"
]
},
"sync_mode": "full_refresh",
"destination_sync_mode": "overwrite"
}
]
}
The configured catalog declares the sync modes supported by the stream (full refresh or incremental). See the catalog guide for more information.
Let's define the stream schema in source-exchange-rates-tutorial/source_exchange_rates_tutorial/schemas/rates.json
You can download the JSON file describing the output schema with all currencies here for convenience and place it in schemas/.
curl https://raw.githubusercontent.com/airbytehq/airbyte/master/docs/connector-development/tutorials/cdk-tutorial-python-http/exchange_rates_schema.json > source_exchange_rates_tutorial/schemas/rates.json
We can also delete the boilerplate schema files
rm source_exchange_rates_tutorial/schemas/customers.json
rm source_exchange_rates_tutorial/schemas/employees.json
As an alternative to storing the stream's data schema to the schemas/ directory, we can store it inline in the YAML file, by including the optional schema_loader key and associated schema in the entry for each stream. More information on how to define a stream's schema in the YAML file can be found here.
Reading from the source can be done by running the read operation
python main.py read --config secrets/config.json --catalog integration_tests/configured_catalog.json
The logs should show that 1 record was read from the stream.
{"type": "LOG", "log": {"level": "INFO", "message": "Read 1 records from rates stream"}}
{"type": "LOG", "log": {"level": "INFO", "message": "Finished syncing rates"}}
The --debug flag can be set to print out debug information, including the outgoing request and its associated response
python main.py read --config secrets/config.json --catalog integration_tests/configured_catalog.json --debug
Next steps
We now have a working implementation of a connector reading the latest exchange rates for a given currency. We're however limited to only reading the latest exchange rate value. Next, we'll enhance the connector to read data for a given date, which will enable us to backfill the stream with historical data.