SFTP Bulk Source Configuration Issue with Filename Filter


The user is facing an issue configuring an SFTP Bulk source to ingest two types of files separately by passing a filename filter in the ‘Globs’ field, but the field is only accepting the generic ‘**’ pattern. The user prefers to separate the data at the source level instead of during the refinement process.


Hey guys!
I have a problem configuring an SFTP Bulk source.

On my server there are two types of files together at the root, but I would like to ingest them separately into two streams.
In the SFTP source settings, I should be able to pass a filename filter in the optional “Globs” field, but it is not accepting anything other than the generic “**” pattern which ends up reading everything and mixing two different data schemes. I didn’t want to have to deal with the separation of this raw data only in the refinement process, as I would already be able to resolve it when reading the source.

Could anyone help me with this?

This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want
to access the original thread.

Join the conversation on Slack

["sftp-bulk-source", "filename-filter", "globs-field", "data-ingestion", "separate-data"]