Search Knowledgebase

Follow

Download XMLs From ICISDA Dataflow to Correct ICIS Data.

Retrieve ICIS-NPDES data from the 45 ICISDA (ICIS Data Access) web services at EPA's Exchange Network Services Center (or through your own node software, OpenNode2, etc.). You could then use the data for your agency's system (performing quality assurance, seeding Exchange Network plugins with data, populating your NPDES or CAA systems, deriving anticipated DMRs for eDMR systems) or resubmit XMLs back into ICIS using the EDT batch process (ICIS-NPDES dataflow) after performing find/replace and other edits.  You could also save retrieved XMLs for testing or training purposes and submit them to the ICIS Test or Stage environments after a data refresh to restore your data. 

 

Registration

To download from the ICISDA dataflow, you will need to register for a CDX account, if you don't have one already, through the Request an Account link on the above ENSC homepage and choose ICISDA for the question "What DataFlow will you primarily be submitting to?". 

 

Hours of Operation

ICIS Nightly Processing begins around 11PM EST when Prod ICIS data is replicated to ICISCOPY, among other processes.  Therefore, please do not query or schedule jobs from 11PM to 7AM EST, when the jobs are usually completed.

 

Test Environment

You may also register for the Exchange Network Test site: https://enservicestest.epacdxnode.net, where the Test ICISDA services connect to the ICIS Stage database instead of ICISCOPY (ICISDA Production connects to ICISCOPY, which is refreshed with ICIS Production data nightly).  ICIS Stage on the other hand is usually refreshed with Production ICIS data about every quarter.  Test ICISDA can be used to retrieve test or training records that you want to save before a data refresh/wipeout to ICIS Stage, which can then be submitted to ICIS Stage using the ICIS-NPDES dataflow on NGNTest2.0 after the refresh.  Saving these templates and modifying them for future use can save a lot of time instead of manually reentering them in the ICIS Stage website.

For guidelines on retrieving NetDMR training/test data from ICIS Stage (before a database refresh), visit: https://netdmr.zendesk.com/hc/en-us/articles/209616306-ICIS-Data-Access .

 

Documentation

The Flow Configuration Document has technical details of the ICISDA dataflow.  This can be found at http://www.exchangenetwork.net/data-exchange/icis-data-access/ .

Here is the list of the 38 services that return XML results that generally conform to the ICIS-NPDES XML Schema format and 7 services that return XMLs with common ICIS ref table codes and descriptions.

ICISDA services returning ICIS-NPDES data families ICISDA services returning ICIS reference tables
GetICISBasicPermitData
GetICISBiosolidPermitData (deprecated version as of 2018)
GetICISBiosolidProgramReportData
GetICISCAFOAnnualReportData
GetICISCAFOPermitData
GetICISComplianceMonitoringData
GetICISComplianceScheduleData
GetICISCSOEventReportData
GetICISCSOPermitData
GetICISDischargeMonitoringReportData
GetICISDMRViolationData
GetICISEnforcementActionMilestoneData
GetICISFederalComplianceMonitoringData
GetICISFormalEnforcementActionData
GetICISGeneralPermitData
GetICISInformalEnforcementActionData
GetICISLimitSetData
GetICISLocalLimitsReportData
GetICISMasterGeneralPermitData
GetICISNarrativeConditionScheduleData
GetICISParameterLimitsData
GetICISPermitTrackingEventData
GetICISPermittedFeatureData
GetICISPOTWPermitData
GetICISPretreatmentPerformanceSummaryData
GetICISPretreatmentPermitData
GetICISScheduleEventViolationData
GetICISSingleEventViolationData
GetICISSSOAnnualReportData
GetICISSSOEventReportData
GetICISSSOMonthlyEventReportData
GetICISStormWaterConstructionPermitData
GetICISStormWaterEventReportData
GetICISStormWaterIndustrialPermitData
GetICISStormWaterMS4LargePermitData
GetICISStormWaterProgramReportData
GetICISStormWaterMS4SmallPermitData
GetICISUnpermittedFacilityData
GetICISAffiliationModuleCrossReferences
GetICISParameters
GetICISPollutants
GetICISUnits 
GetICISUnitConversions
GetICISUnitGroups
GetICISUnitGroupCrossReferences

 

For more information about the ICIS-NPDES dataflows (EDT Batch submissions), visit: https://icis.zendesk.com/hc/en-us/articles/207065186-Guide-to-submitting-dataflows-to-ICIS

Presentation from a 2016 ICIS-NPDES monthly call with some instructions for sample service requests.

 

Hints and Tips

For string parameters (not dates, integers, etc.), the string can be partial matches, e.g., ‘TX’, ‘tx’ (case is insensitive), or ‘TX003’ for NPDESPermitNmbr, searches for all occurrences of this substring within the parameter (‘%tx%’ will also search for ‘tx’ anywhere in the string, whereas ‘tx%’ will search for the occurrence of this term only in the beginning of the string).

Code-type parameters (usually ending with ‘Code’ and held in ICIS ref code tables) are programmed to accept multiple values as input, but must be separated by commas with or without spaces (this parameter input is passed as a string to the SQL IN operator).  The NPDESPermitNumber field can also include comma separated values if the service is a solicit-type service listed in Section 3.3 of the FCD (they must be complete NPDES IDs, not wildcards).

Older data returned from ICIS might not conform exactly to the current Schema, so if the intent of the user is to submit the XML back into ICIS, the user might need to edit the non-conforming data.  For example, if a ScheduleViolationCode is returned in the GetICISComplianceScheduleData service, it might contain ‘C10’ or ‘C20’ values from legacy PCS, but the ICIS Schema only accepts either ‘C30’ or ‘C40’.  Another example is that some <EnforcementActionGovernmentContact> blocks returned in the GetICISFormalEnforcementActionData service, might not contain any <ElectronicAddressText> data, which is required by the ICIS XML Schema.  

Similarly, ICISDA mostly returns the raw data values stored in the ICIS database, which might not conform to the ICIS Schema.  For example, the <NumberOfReportUnits> values returned in GetICISBiosolidProgramReportData could be ‘5’, but the ICIS Schema wants ‘05’, so it is up to the user to convert the format to whatever system they are working with- their own system or ICIS or something else. The only exceptions to this raw data return are the values in the PermitStatusCode and StatusCode tags for Basic and General Permits as detailed in the FCD.

The <Header> tag at the beginning of all returned XMLs will have an <Id> tag value of ‘ICISDA’.  Since this user ID does not exist in ICIS, the user must change these generic values to their specific, registered user ID, if they want to submit the XML back into ICIS.  Also, if you intend to submit the XML to ICIS, you should include 'Source' and 'FullBatch' in the <name> and <value> tags respectively in the <Header> so that ICIS EDT returns a Response PDF file with details of your submission.  For example:

<Header>
  <Id>YourEDTUserID</Id>
  <Property>
    <name>Source</name>
    <value>FullBatch</value>
  </Property>
</Header>

GetICISParameterLimitsData includes a parameter called IncludeAllVersions, which defaults to False if not entered.  Enter 'true' (case insensitive) if you want to return records from previous versions of the permit.

Four other services return data from ALL versions of a permit by default, including previous versions: GetICISDischargeMonitoringReportData, GetICISDMRViolationData, and GetICISScheduleEventViolationData, GetICISSingleEventViolationData.  The rest return only the current version of permits.

For federal compliance monitoring data, use the GetICISFederalComplianceMonitoringData (State_EPA_Flag = 'E') instead of GetICISComplianceMonitoringData (State_EPA_Flag = 'S').

Was this article helpful?
0 out of 0 found this helpful
Have more questions? Submit a request