Challenge Overview
Federal Aviation Administration Notice to Airmen (NOTAM) API Architecture Challenge
The Federal Aviation Administration (FAA) in collaboration with Topcoder is working to further improve timely access to Notice to Airmen (NOTAM) data by developing required Application Programming Interfaces (APIs) to meet the diverse needs of the aviation industry.
A Notice to Airmen (NOTAM) is the real-time notification component of the FAA's Aeronautical Information System and contains up-to-date information related to any change in the National Airspace System (NAS).
Who uses NOTAMs?
Challenge overview:
Welcome to the NOTAM API Architecture Challenge!
Background Information
NOTAM information is currently published via the NOTAM Distribution Service, using two different methods: FIL (FNS Initial Load) and Java Messaging Service - JMS.
Temporary flight restrictions – TFRs
Aeronautical Information Portal
Aeronautical Information eXchange Model (AIXM) 5.1:
The system that we’re building will need to ingest all the data sources and provide an API to query all the data based on various properties, such as:
API Requirements:
API Access:
Task Details
Your task is to analyze the system requirements and propose a system architecture that specifically includes:
It is up to you to suggest how best to handle different data formats and querying of the large number of records in a scalable way. For example you can discuss these points:
In summary the architecture should consider, ingestion of NOTAM data, synchronization internally to ensure all data is available, consumption of data from TFR and SAA websites, merging and integrating with the primary NOTAM data set. Then convert to various formats and deliver it in a performant manner to support a large number of users in a concurrent fashion. The architecture should also be configurable to changes, so when new changes or new NOTAM types come in, the architecture facilitates easy updates, rather than hard-coded design.
Also, discuss details of merging the TFR and SAA data with primary NOTAMs data from NDS (FIL and JMS).
Note that the system should not rely on any products specific to one cloud provider, but should be deployed to local clusters too. That makes products like DynamoDB inadequate, but using products such as Elasticsearch, mongo, postgres is ok.
Your submission should be a document that details the system architecture. Using charts/diagrams is highly encouraged, however reasoning for the proposed architecture is required - it is not enough to just provide diagrams without further explanation.
The Federal Aviation Administration (FAA) in collaboration with Topcoder is working to further improve timely access to Notice to Airmen (NOTAM) data by developing required Application Programming Interfaces (APIs) to meet the diverse needs of the aviation industry.
A Notice to Airmen (NOTAM) is the real-time notification component of the FAA's Aeronautical Information System and contains up-to-date information related to any change in the National Airspace System (NAS).
Who uses NOTAMs?
- Commercial, General Aviation, and Drone pilots need access to NOTAM data before and during flight.
- The FAA issues NOTAMS to support multiple Federal Agencies, Airports, protect critical infrastructure, disaster relief efforts and space operations.
Challenge overview:
Welcome to the NOTAM API Architecture Challenge!
- The purpose of this challenge is to develop a modern set of APIs for the FAA to standardize and improve access to the safety-critical NOTAM data for all consumers.
- Your task in this challenge is to design the architecture for the NOTAM API system - specify system components, technologies, frameworks/libraries and data flow between the components.
Background Information
NOTAM information is currently published via the NOTAM Distribution Service, using two different methods: FIL (FNS Initial Load) and Java Messaging Service - JMS.
- FIL provides a snapshot of all NOTAMs at a point in time - data is provided via SFTP.
- JMS provides updates to existing NOTAMs and new NOTAMs.
- Fil and JMS should be combined to build a complete and up to date datastore of NOTAMs.
- For example, when you start consuming data from JMS you only get data from that point onwards - in order to ensure we have the complete data, first we need to get the initial data from FIL - which is the data that was published prior to the time we started consuming the data from JMS. Also, FIL data should be fetched periodically to ensure no NOTAMS are out of sync with NDS datastore.
Temporary flight restrictions – TFRs
- Defines a temporary restricted airspace/area due to a hazardous condition, special event or other conditions that restrict the use of the airspace.
- TFRs are present in the NOTAMs published by the Distribution Service, but they often lack geometry information, so all records will need to be pulled from the TFR website directly - https://tfr.faa.gov/tfr2/list.html, and merged with the NOTAM data from the NOTAM Distribution service.
Aeronautical Information Portal
- Contains static data that relates to NOTAMS - Airport, Runway, Air Route Traffic Control Centers (ARTCC), NAVAID, Special Activity Airspace (SAA) information. Data is published on this site every 28 days and is available as a zip file (NASR subscription) –
- This data needs to be merged with the base NOTAM data from the NOTAM Distribution service - merging the geometry data, similar to how the TFR geometry data is used to enrich the base NOTAM data.
Aeronautical Information eXchange Model (AIXM) 5.1:
- NOTAM data is provided in a specific XML based format called Aeronautical Information eXchange Model (A IXM) 5.1 - this data is provided by the NOTAM Distribution Service (both FIL and JMS will provide AIXM 5.1 format)
The system that we’re building will need to ingest all the data sources and provide an API to query all the data based on various properties, such as:
- Location
- NOTAM type
- Location and NOTAM number
- Effective start and end dates
- Affected feature types and;
- Last updated date.
API Requirements:
- Provide a parameter to specify an output data format - either AIXM 5.1, AIDAP, GeoJSON or unmodified.
- Will need to convert the source NOTAM data (AIXM 5.1) between AIXM 5.1, GeoJSON and AIDAP.
- AIDAP is a legacy format, for which we have a direct field mapping from AIXM.
- Scale horizontally to support a large number of API calls.
- 2-3 million NOTAMS are issued per year and updated on a continuous basis.
- Data should be consumed and available as soon as it is published.
- It is expected that there will be more reads for data than updates from the original source.
API Access:
- Users must register on a developer portal to get an access key for the API (building this portal is in scope of the project)
- All application components are hosted on Kubernetes and will need to be dockerized.
Task Details
Your task is to analyze the system requirements and propose a system architecture that specifically includes:
- Identification and documentation of system components (for example data ingestion, api, developer portal)
- Definition of technology stacks for each of the components (databases, frameworks, and any other details)
- Definition of data flow between all of the system components (what component updates which data, where are the updates stored, what happens with deleted data and how the system ensures the API data store has all the data, i.e. merging FIL and JMS data)
It is up to you to suggest how best to handle different data formats and querying of the large number of records in a scalable way. For example you can discuss these points:
- Should all the records be converted to all supported data formats during ingestion, or only when data is returned by the API?;
- How to handle querying the xml data - do we extract the supported query fields during extraction and store them separately for querying purposes?
- Does data ingestion limit the choice of programming language/frameworks (ex we need to read NOTAM data from JMS queue)
- Do we need a relational database? Or an analytics stack like Elasticsearch?
- Do we need a caching layer (ex Redis)
- Do we need a message queue for data ingestion (for example Kafka/RabbitMQ)?
In summary the architecture should consider, ingestion of NOTAM data, synchronization internally to ensure all data is available, consumption of data from TFR and SAA websites, merging and integrating with the primary NOTAM data set. Then convert to various formats and deliver it in a performant manner to support a large number of users in a concurrent fashion. The architecture should also be configurable to changes, so when new changes or new NOTAM types come in, the architecture facilitates easy updates, rather than hard-coded design.
Also, discuss details of merging the TFR and SAA data with primary NOTAMs data from NDS (FIL and JMS).
Note that the system should not rely on any products specific to one cloud provider, but should be deployed to local clusters too. That makes products like DynamoDB inadequate, but using products such as Elasticsearch, mongo, postgres is ok.
Your submission should be a document that details the system architecture. Using charts/diagrams is highly encouraged, however reasoning for the proposed architecture is required - it is not enough to just provide diagrams without further explanation.