Teach a PDF parser where each field sits in a work order
A PDF parser reads the text layer of a PDF. You open a sample PDF from the sender, mark which piece of text is the order number, which is the site address and so on, and the import reads the same positions in every PDF that arrives. This page only exists for a parser whose Document type is PDF, and the parser has to be saved on Properties first.
The page has four tabs: Reference documents, Field rules, Guard rules and Test. The Test tab only shows once the parser is saved.
Reference documents
A reference document is a sample PDF the rules are built on. A parser needs at least one, and each one keeps its own rules and its own guard rules.
- Open Reference documents and drop a PDF on Add a reference PDF. The drop zone takes PDF files only.
- The page moves to Field rules with the PDF's text listed beside the PDF itself.
- Mark the fields (below), then click Save. Saving uploads the PDF and stores its rules.
Click a saved document to open its rules again. Delete appears under each document once there is more than one, with the prompt "Are you sure you want to remove this reference document and corresponding rules?". The last document can't be deleted. A delete is immediate and doesn't wait for Save.
Add a second document when the same sender uses more than one layout. Every document's rules are tried on each incoming PDF in the order they were saved, and the first one that finds a value for a field supplies it.
A parser imported from an export file carries its rules without the PDF they were built from. Its document shows as Imported rules, greyed out, and can't be opened. Add a PDF to build on.
Leaving the page, reloading it, or opening another reference document while rules are applied but unsaved brings up "Unsaved changes". Leave without saving and Open without saving discard them.
Field rules
Each row on the left is one piece of text from the PDF. Hover a row and the PDF marks it. Click the three dots button on a row to choose what it fills in, from two lists: Direct fields and Data points. A coloured chip on the row shows what the text is used for. The gear on a chip opens its options and the cross removes it.
One job or Several jobs
One job is a work order with one job on it, and it is where most parsers live. Several jobs is for a PDF that lists jobs or tasks in a table. See Several jobs in one PDF below.
Direct fields
A direct field takes the text at the position you marked.
| Pick | What it reads |
|---|---|
| Order Number, Project Name, Project Description | The text at the marked position. |
| Date Issued, Due Date, Due Time, Response Time | Simpro only. A Fergus parser isn't offered them. |
| Site Name, Site Address | The site. |
| Site Address: Street, City, State, Post code | The address in parts. Picking one clears Site Address, and picking Site Address clears all four. |
| Customer Contact: Name, Phone, Mobile, Email | A single customer contact. |
| Site Contact: Name, Phone, Mobile, Email, Position | A single site contact. Fergus is offered Position. |
| Site Contact: Department, Notes | Simpro only. |
| Cost Centre Total: Ex Tax, Inc Tax | Simpro only. |
| Project Description: Start and End | Everything between two labels. See below. |
| Site contacts: Start and End, Customer contacts: Start and End | A block of people between two labels. See below. |
| [Custom]: name (type) | Simpro only. Listed under Custom fields from the custom fields in your Work Order Import settings. |
A field has one position. Marking the same field on another piece of text moves it. Project Description and the Start and End pair are alternatives, and picking one removes the other.
Project Notes isn't in the direct list. It is only offered as a data point.
Data points
Data points collects text from several places into one field. It is offered for Project Name, Project Description and Project Notes only. Each text you mark adds to the field: on a new line it is joined with a line break in a description or notes and with " - " in a name, and on the same line with a space. A data point can also be inserted by name into a template.
The gear on a data point has Reference name for template, the two scan options, and Text to match and Text to ignore (see When a value is wrong).
Description and contact blocks
Mark a Start label and an End label and the import reads everything between them, in reading order, across pages. Page furniture such as "Page 1 of 2" and text repeated at the same position on more than one page is dropped. The Start and End text themselves are left out.
If the End label isn't found, nothing is read. The import doesn't guess where the block stops.
A block of contact details works the same way. Each person in it becomes a contact on the site or the customer:
- A line with no phone number or email starts a new person, and the lines under it are theirs.
- Labels such as (m), (h), (w), (e), "Mobile:" and "Phone:" are understood, and values such as n/a, nil, none and TBC are ignored.
- A phone number needs at least eight digits.
- A person with no name and no way to reach them is dropped.
On a page with two columns, set Width limit (optional) and Height limit (optional) under Reading area in the options of the Start label. They are in PDF points, and an A4 page is 595 wide and 842 high. A block that sits in the right hand column also stops at the next heading on its row.
Tick Use the Site contact: email field as the email of the primary site contact when a work order lists its site contacts as a block and gives one email address somewhere else. That email is put on the first person in the block. It only applies when no single site contact name, phone or mobile was read. The tick box saves as soon as you click it, and doesn't wait for Save.
Field options
The gear on a field opens Field parsing options.
| Option | What it does |
|---|---|
| Scan X-Coordinate when exact coordinate is not found. (vertical) | Looks down the same column of the page. |
| Scan Y-Coordinate when exact coordinate is not found. (horizontal) | Looks along the same row. |
| Field label - used for scanning values on page | The label the value sits next to. |
| Secondary field labels (optional) | Other labels the value might have. Tried when the first label isn't found. |
| Field label position - used for identifying values | None, Left (the label is on the same row, to the left) or Top (the label is above, in the same column). |
| Horizontal alignment | Left, Centre or Right. Tells the scan how the text lines up. |
| Vertical alignment | Top, Middle or Bottom. |
The two scan options are exclusive, and ticking one clears the other.
An exact match means the text sitting within one PDF point of the position you marked. When one text is there, it is the value. When several are, the nearest to the marked height wins. When nothing is there, the scan options are tried. With Scan X and no field label, the nearest text in the column is taken, and with Scan Y and no label, the nearest text in the row.
A field with a Field label set can also be found on another page. A longer block earlier in the document pushes text onto the next page, so the import tries the marked page first and the rest of the PDF after.
Suggest rules
Suggest rules fills in the common fields from the labels in the open PDF, and shows what it found before it adds anything.
| It looks for | Using labels such as |
|---|---|
| Order number | Job number, Work order number, Purchase order, PO number, Reference number |
| Project name | Summary, Job summary, Job title, Title, Subject |
| Site address | Property, Site address, Job address, Address, Location, Premises |
| Date issued | Created, Date created, Issued, Order date, Date raised |
| Due date | Due, Due date, Required by, Complete by, Target date |
| Project description | A heading such as Description or Scope of works, up to a section such as Notes or Attachments |
| Site contacts, Customer contacts | A heading such as "For access contact the tenant on" or "For queries contact the agent on", followed by people with a phone number or email |
The table lists each suggestion with the Field, Found next to, Value found, How it will be found and Existing rule columns. Tick the ones you want and click Apply N rules. A suggestion for a field that already has a rule starts unticked and shows Replaces. For a list of people, the Field column is a dropdown that decides whether they are added as site contacts or customer contacts.
Applied suggestions are on screen only. Click Save to keep them. When the value and its label are in one piece of text, such as "Job number - 16171", the suggestion also adds a filter that removes the label. That filter is stored with the parser's filters and needs a save on Field filters.
A single contact, times and totals aren't suggested, because they vary too much between senders. Map them by hand.
Several jobs in one PDF
Switch to Several jobs when one PDF lists several jobs, such as the rows of a table. Click Add reference and describe each repeating column in Multi-job reference options.
| Option | What it does |
|---|---|
| Field | Order number, Project name, Project description, Project notes or one of your template fields. |
| Reference text | The heading of the column, picked from the PDF. It sets where the column sits. |
| Horizontal alignment | Left, Centre or Right, to match how the column lines up. |
| Field label reference for finding values | Required on the primary reference. Regular expressions are allowed. Only text that matches is taken. |
| Field label position | None, Left or Top. |
| Capture all rows below first label match and Stop label | With Left only. Keeps taking rows until the stop text appears in the label column. |
| Text to ignore | Regular expressions. Text that matches is skipped. |
| Pre-text to add before PDF value (optional) | Added in front of each value. |
| Primary reference field (only one allowed) | The column that has one entry per job. |
| Add job count to sub-jobs | Adds " - Job 1", " - Job 2" to the name. Needs a primary reference. |
| Multiple job save mode | Multiple jobs or Single job. |
| Ignore saving master job when a single job is listed | With Multiple jobs only. |
Each text in the primary column starts a job, and the text in the other columns down to the next primary text belongs to it. Marking a new primary reference clears the primary flag on the others. The table on the page shows Field, String reference, Primary and Save mode for each reference, with Manage and Delete.
In Multiple jobs mode a Simpro import makes the main job and then one more job or quote for each row, copying the main one and replacing the order number, name, description and notes. In Single job mode the rows are joined into the one job, with " / " between values and line breaks in descriptions and notes.
A Fergus import doesn't make extra jobs from the rows. Use Single job on a Fergus parser.
Guard rules
Guard rules say which PDFs the parser is for. A PDF is only processed when it contains all of the guard text, so a work order that looks the same but belongs to another business, branch or inbox is skipped. Guard rules belong to the open reference document. With none, every PDF is processed.
- Open the Guard rules tab with a reference document open.
- Type text under Add text and click Add, or search under Or pick text from the reference PDF and click Use as guard. Pick something that doesn't change from one work order to the next, such as the contractor's name or a licence number.
- Choose Contains or Is exactly for each rule.
- For text picked from the PDF, tick Only near where it sits to look for it on the same page within about 40 PDF points of where it was in the sample.
- Click Save.
Case and spacing are ignored. Contains finds the text in a single piece of text, a line across the page, or the page read in order. Is exactly needs a single piece of text or a line to be the text and nothing else.
A PDF passes when it meets every rule of at least one reference document that has guard rules. A reference document with no guard rules isn't counted. If one document has guard rules and another doesn't, a PDF that fails the first is still skipped.
A PDF that fails is skipped and recorded on the Dashboard with "Skipped: The PDF did not match the parser's guard rules" and the text it was missing. If the inbox has recipients under When a work order was not processed, they get an email headed "Work order skipped", with the original PDF attached when Attach the original PDF is on. See Inbox rules.
Test
Test runs the saved rules on any PDF and shows what they extract. Nothing is created in Simpro or Fergus, no inbox record is written, and the usage meter isn't touched.
- Open Test and drop a PDF on the drop zone.
- Read the Extracted fields table. Each field shows its value as the import would have it before it is written.
- If the parser has guard rules, a Matches the guard rules or Does not match the guard rules badge shows. A PDF that fails is still read, and the table is shown, but an import would skip it.
The test uses what is saved. Rules you have applied and haven't saved aren't in it.
How it maps to Simpro
| Parsed value | Simpro job or quote |
|---|---|
| Order Number | Order number |
| Project Name | Name |
| Project Description | Description |
| Project Notes | Notes |
| Date Issued, Due Date | Date issued and due date. Read as YYYY-MM-DD or D/MM/YYYY. Anything else is left out. |
| Due Time | Due time on a job. A quote has no due time. |
| Response Time | Matched against the inbox's response times by text. A quote has no response time. |
| Site Name, Site Address, contacts | The site and its contacts, placed using the inbox's settings. |
| [Custom] fields | The matching job custom field. |
How it maps to Fergus
A Fergus job takes the Order Number as its customer reference, the Project Name as its title and the Project Description as its description. A blank title becomes "Work order" followed by the order number, and a blank description becomes the title. Project Notes become a note on the job. A Fergus parser can't read dates, times, response times, department, site contact notes, cost centre totals or custom fields. See Destination fields.
When it goes wrong
A field is empty on every import
The marked position didn't line up. The import only takes text within one PDF point of where you marked, so a sender whose layout shifts between work orders needs a scan option, or a field label with its position. Test the parser on a recent PDF to see which fields come back.
A field is empty and it has a Field label
With a Field label set and Field label position left on None, the import has no direction to check the label in, and takes no value. Set Left or Top.
A scan picks up the wrong text
With Scan X or Scan Y and no Field label, the import takes the nearest text. With a label set and not found, it takes nothing. Set a label and its position.
Text to match or Text to ignore doesn't behave
Text to match and Text to ignore on a data point are matched as regular expressions, ignoring case, against each piece of text at the marked position. Text that matches Text to ignore is dropped. When Text to match has entries, only text that matches one of them is kept.
"No PDF Attachments in email"
The email has no attached PDF. A PDF parser reads files attached to the email as PDFs. A PDF linked in the body of an email is handled by an email parser that points at a PDF parser. See Email parser rules.
"No fields were extracted from this PDF. Check the parser settings."
The rules found nothing in the PDF. Check the PDF has selectable text. The parser reads the text layer, and a scanned image has none.
Site details not mapped
The badge beside the parser name means no rule reads a site name or a site address. Add one, or the import has no site to place the job on.
Save is missing
Save shows on Field rules and Guard rules once a reference document is open. It also needs the parser to be saved already, since Add a reference PDF only appears after that.