Retrieve ICIS-NPDES data from the 45 ICISDA ( ICIS Data Access ) web services at EPA's Exchange Network Services Center (or through your own node software, OpenNode2, etc.). You could then use the data for your agency's system (performing quality assurance, seeding Exchange Network plugins with data, deriving anticipated DMRs for eDMR systems) or resubmit XMLs back into ICIS using the EDT batch process (ICIS-NPDES dataflow) after performing find/replace and other edits. You could also save retrieved XMLs for testing or training purposes and submit them to the ICIS Test or Stage environments after a data refresh to restore your data.
To download from the ICISDA dataflow (ICIS Production data), you will need to register for a CDX account, if you don't have one already, through the Request an Account link on the above ENSC homepage and choose ICISDA for the question "What DataFlow will you primarily be submitting to?".
Hours of Operation
ICIS Nightly Processing begins around 11PM EST when Prod ICIS data is replicated to ICISCOPY, among other processes. Therefore, please do not query or schedule jobs from 11PM to 7AM EST, when the jobs are usually completed.
For manual users of ICISDA, login to the Exchange Network Services Center at https://enservices.epa.gov. Within the Exchange Network Services tab, search for any of the services listed in the Documentation section below, or simply search 'ICISDA', which will list them all. The Production ICISDA services have a Dataflow value of ICISDA and a Node value of .NetNode2 (see highlights below). ICISDA Production retrieves data from the ICIS Copy database, which in turn is a nightly copy of data from the ICIS Production database.
You may also register for the Exchange Network Test site: https://enservicestest.epacdxnode.net, where the Test ICISDA services connect to the ICIS Stage database (not ICIS Test). Whereas the data in ICIS Copy is refreshed nightly, ICIS Stage, is usually refreshed with Production ICIS data about every three months. Test ICISDA can be used to retrieve test or training records that you want to save before a data refresh/wipeout to ICIS Stage, which can then be submitted back to ICIS Stage using the inbound ICIS-NPDES dataflow on the .NGNTest2.0 Node (not .NGNDev2.0. See screenshot below) after the refresh. Saving these templates and modifying them for future use can save a lot of time instead of manually reentering them in the ICIS Stage website.
For guidelines on retrieving NetDMR training/test data from ICIS Stage (before a database refresh), visit: https://netdmr.zendesk.com/hc/en-us/articles/209616306-ICIS-Data-Access .
The Flow Configuration Document has technical details of the ICISDA dataflow. This can be found at http://www.exchangenetwork.net/data-exchange/icis-data-access/ .
Here is the list of the 38 services that return XML results that generally conform to the ICIS-NPDES XML Schema format and 7 services that return XMLs with common ICIS ref table codes and descriptions.
|ICISDA services returning ICIS-NPDES data families||ICISDA services returning ICIS reference tables|
GetICISBiosolidPermitData (deprecated version as of 2018)
For more information about the inbound ICIS-NPDES dataflows (EDT Batch submissions), visit: https://icis.zendesk.com/hc/en-us/articles/207065186-Guide-to-submitting-dataflows-to-ICIS .
Presentation from a 2016 ICIS-NPDES monthly call with some instructions for sample service requests.
Hints and Tips
For string parameters (not dates, integers, etc.), the string can be partial matches, e.g., ‘TX’, ‘tx’ (case is insensitive), or ‘TX003’ for NPDESPermitNmbr, searches for all occurrences of this substring within the parameter (‘%tx%’ will also search for ‘tx’ anywhere in the string, whereas ‘tx%’ will search for the occurrence of this term only in the beginning of the string).
Code-type parameters (usually ending with ‘Code’ and held in ICIS ref code tables) are programmed to accept multiple values as input, but must be separated by commas with or without spaces (this parameter input is passed as a string to the SQL IN operator). The NPDESPermitNumber field can also include comma separated values if the service is a solicit-type service listed in Section 3.3 of the FCD (they must be complete NPDES IDs, not wildcards).
Older data returned from ICIS might not conform exactly to the current Schema, so if the intent of the user is to submit the XML back into ICIS, the user might need to edit the non-conforming data. For example, if a ScheduleViolationCode is returned in the GetICISComplianceScheduleData service, it might contain ‘C10’ or ‘C20’ values from legacy PCS, but the ICIS Schema only accepts either ‘C30’ or ‘C40’. Another example is that some <EnforcementActionGovernmentContact> blocks returned in the GetICISFormalEnforcementActionData service, might not contain any <ElectronicAddressText> data, which is required by the ICIS XML Schema.
Similarly, ICISDA mostly returns the raw data values stored in the ICIS database, which might not conform to the ICIS Schema. For example, the <NumberOfReportUnits> values returned in GetICISBiosolidProgramReportData could be ‘5’, but the ICIS Schema wants ‘05’, so it is up to the user to convert the format to whatever system they are working with- their own system or ICIS or something else. The only exceptions to this raw data return are the values in the PermitStatusCode and StatusCode tags for Basic and General Permits as detailed in the FCD.
The <Header> tag at the beginning of all returned XMLs will have a default <Id> value of ‘ICISDA’. If you plan to edit and submit the XML back into ICIS, replace this with your EDT approved ICIS User ID. The <Property> block is included with the values 'Source' and 'FullBatch' in the <name> and <value> tags so that ICIS EDT returns a Response PDF with the Acceptance and Rejection details of your submission. For example:
<Id>ICISDA</Id> <!-- Replace with your EDT User ID -->
GetICISParameterLimitsData includes a parameter called IncludeAllVersions, which defaults to False if not entered. Enter 'true' (case insensitive) if you want to return records from previous versions of the permit.
Four other services return data from ALL versions of a permit by default, including previous versions: GetICISDischargeMonitoringReportData, GetICISDMRViolationData, and GetICISScheduleEventViolationData, GetICISSingleEventViolationData. If you do not want data from previous permit versions, please enter the appropriate Begin and End Dates to create your date range for Monitoring Period End Date (MPED), Schedule Event Date, or Single Event Violation Date. The rest of the services return only the current version of permits.
For federal compliance monitoring data, use the GetICISFederalComplianceMonitoringData (State_EPA_Flag = 'E') instead of GetICISComplianceMonitoringData (State_EPA_Flag = 'S'), which returns state compliance monitoring data.