AWS SWF / Step Functions Expert

General information for the business: The system polls aircraft flight logs website apis from two different websites Description of requirements/functionality: Please take a look and let me know if: a) This is possible to do using AWS SWF or Step Functions? b) If yes, would you choose SWF or Step Functions to do this? c) Is this something you can do? If so how long would it take? Overview of the system: 1. The system polls aircraft flight logs website apis from two different websites to see details of new files. 2. If there are new files, they are downloaded to S3. The data arrives in a variety of different formats depending on the aircraft type. 3. The data files are compressed using the appropriate utility. 4. The data is normalized and converted into a .ofd file. 5. Afterwards all files are standardized and converted to a *-cp.ofd format. 6. Finally a list s-expr file (ascii text) is built from the ofd and cp.ofd files and is uploaded to a database. Currently, 1 and 2 are performed by a shell script that runs in the background on one of the servers. Steps 3-6 are performed by Python scripts handle the process flow and call shell scripts to do the actual processing. Database tables are updated to control the state flow of some of the processes, for example, if there is an error normalizing the data in step 4, then a record in the database will be updated to "error". When there are large amounts of data to be processed, the database becomes the bottleneck, hence the requirement to move to SWF/Step Functions to control the process. Obviously a database is required somewhere in this process, but it can be new (DynamoDb) to manage these state changes. Also, can SWF/Step Functions handle these processes without a database? I have all the scripts. And how many hours do you need for this project? Let me know what you think and please ask any questions you may have. OS requirements: Windows, Linux, Other Extra notes: