Dominion Voting Systems - Patents Ip Binder

111111

(12)

(54)

United States Patent

(10)

Poulos et al.

(45)

SYSTEM, METHOD AND COMPUTER PROGRAM FOR VOTE TABULATION WITH AN ELECTRONIC AUDIT TRAIL

(75) Inventors: John Poulos, Toronto (CA); James Hoover, Montreal (CA); Nick Ikonomakis, Toronto (CA); Goran Obradovlc, Toronto (CA) (73) Assignee: Dominion Voting Systems (CA) ( *)

Notice:

Subject to any disclaimer, the term of this patent is extended or adjusted under 35 U.S.C. 154(b) by 1320 days.

(21) Appl. No.: 11/121,997 (22) Filed:

MayS, 2005

(65)

Prior Publication Data US 2005/0247783 Al

(30)

1111111111111111111111111111111111111111111111111111111111111 US008195505B2

Nov. 10,2005

Foreign Application Priority Data

May 5, 2004 (51) Int. Cl. G06F 11/00 (52) U.S. Cl.

(CA)

2466466

(2006.01) 705/12; 705/50; 705/64; 705/75; 235/386 (58) Field of Classification Search .................... 705/12, 705/50--79; 235/386 See application file for complete search history.

Patent No.: Date of Patent:

(56)

US 8,195,505 B2 Jun. 5,2012

References Cited U.S. PATENTDOCUMENTS

6,250,548 Bl * 7,406,480 B2 * 2007/0170253 Al *

6/2001 McClureet al. 7/2008 Seibelet al. 7/2007 Chung et al.

235/51 111 235/386

OTHER PUBLICATIONS White, Ron, "How ComputersWork",MillenniumEd., Que Corporation, Indianapolis,IN, 1999.*

* cited by examiner Primary Examiner - Chrystina Zelaskiewicz (74) Attorney, Agent, or Firm - Holland & Hart LLP (57) ABSTRACT A system,method and computer program for tabulating votes and creating an audit trail is provided. A ballot processing device is integrated with a ballot box. The ballot processing device includes a paper feed mechanism, a computer, a ballot processing application loaded on the computer, and a digital scanning device linked to the computer.The ballot processing application provides instructions to the computer to initiate the digital scanning device to scan a paper ballot fed into the paper feed mechanism so as to create a digital image of the paper ballot. The ballot processing application also processes the digital image to establish a series of processing results defining one or more voting results for the paper ballot, and also an audit trail. The ballot processing application processes the digital image to define the voting results based on criteria established by election officials. The audit trail enables election officials to verify that particular paper ballots have been processed correctly in accordance with these criteria. 20 Claims, 7 Drawing Sheets

The voting session begins from the beginning and the voter 'navigales' through Ihe ballol selecting as many candidates as desired The baUot is cancelled and

thevoler returns to the start of the ballol 10 begin

again No

Yes

+ The tabulator makes an audible sound 10

alert the etection officialthat Ihe audio vole has been compteled No

Tabulator is ready for the next ballol

u.s. Patent

Juo.5,2012

Sheet 1 of7

US 8,195,505 B2

Voter Enters Voting Location

Voter deemed ineligible

t

Voter cannot vote

t

Does the voter require an audio or paper ballot?

...---------, Print the required ballot 'on-demand' on the local printer and issue to voter

I

Voter deemed eligible to vote

Is the ballot required by the voter preprinted?

No

ToFig.lb

Audio

Yes Iss~e pret._ printed paper ballot to voter

Voter marks the paper ballot

Electionofficial to placemarkedballot is put into the 'Spoiled Ballots' envelope, Voteris issueda new ballot and is reo instructed on ballot marking

Yes

Ballot feeds completely through the tabulator and falls into the ballot box

t Tabulator to ask user. Are the overvotedl blank races intentional?

Does the tabulator detect anyovervotesor blank or blank races?

Tabulator is ready for next paper ballot

FIG.1a

u.s. Patent From Fig. 1a

Juo.5,2012

Sheet 2 of7

US 8,195,505 B2

Voter is escorted to the audio voting area where he is given a set of earphones and a handheld pendant thumb switch

t

Once the voter is set up, the election official selects the proper audio ballot on the tabulator to initialize the voting session

J The voting session begins from the beginning and the voter 'navigates' through the ballot selecting as many candidates as desired

_.,_

t The ballot is cancelled and the voter returns to the start of the ballot to begin again

At the end of the audio voting process, the voter is prompted to review the selected choices one last time and to confirm the selections

Werethe selections made confirmed by the voter?

No

Yes

t The tabulator makes an audible sound to alert the election official that the audio vote has been completed No The election official assist the voter in leaving the audio voting area

The electionofficial asksthe voter:Are you happywith the balot- do you indeedwish to cast the ballot? The election official prompts the tabulator to accept the ballot, and a paper reciept is printed and falls into the ballot box

FIG. 1b

Tabulator is ready for the next ballot

u.s. Patent

US 8,195,505 B2

Sheet 3 of7

Juo.5,2012

10 22

Paper BalIot Path

~~"-

14

24

~

36 "-:.:.:.:.:.:.:.:.:.:.:.:.:. :::::. ------~ --~--- ----- ---', "

,

:

AUXiliary} ~~~~~

: --~~

:

-------"

Path Auxiliary Ballot Box

Tabulator Ballot Box

12

FIG.2

o

... ,

~16

u.s. Patent

US 8,195,505 B2

Sheet 4 of7

Juo.5,2012

/ ~ Em

~

To vote, completely fill in the box(es)

Em Em

BALLOT FOR THE OFFICE OF

Em

Vote for onlyONE(1) candidate

1) Robert L. t-lLJean

Em

3) Jacques

Em Em

4) Samuel de

Em

6) Leif

Em

7)John

Em Em

8) Henry

Em

10)William L.M.

Em

m

~

~ ~ ~Il~

~

ORO

Em Em

/

5)James

9) Jake

11)Wilfrid

MAYOR

12)John A. 13) lester B. 14)louis S.

Em

15)Pierre E.

Em

I

•

o next to your choice(s), like this: BALLOT FOR THE OFFICE OF

COUNCILLOR Candidate1 DO Candidate2 • D Candidate3 D.

Vote forONE(1) 1st choice & ONE(1) 2ndchoice,as shownat right

BaRDON

"t} '\,~'I::.

CABOT

MASSEY DO 0 1) MilesJ. O'DANIEl DO CHAMPLAIN 0 2) Menelaus P. VON ESPIE DO 3) Clause COOK 0 ERIKSSON 0 <0-103 ~ ~ .:£! se ~ ~ :z '- m s: r*" -n ffi ~ m c: FRANKLIN 0 ~ => ~. Q 0' => ~ '< ....,0 => ....,-1 co' -< => '< c m ;s: HUDSON 0 °b m;:o 8' JOLIET 0 m m KING 0 :9 s: lAURIER 0 0 c. :z a: » 0 -t MacDONALD 0 s: en '" rCD m s: 0 :z 0 r- » PEARSON 0 s: m c: :z » m 0 m ,..... ::;u :z m -< SAINT·lAURENT 0 :s: » TRUDEAU 0 z 000000 10 CARTIER

'T1

III

Il>

Il>

0

"0

-I ::J: ;:0

n III :::l

"'0

(')

G')

.."

0

;:0 0

0

VI

(')

;:0::

'T1

;:0

~

(')

G')

~ ~ ~ ~ ~ ~ ~ f2Zl ~ ~ ~ ~ f2Zl f2Zl f2Zl I2Z3

~

(')

f2Zl

;:0 ;:0

f2Zl ~ f2Zl ~

Em ~

~

~

;:0

Em

,

January 11891

Em

~

~

General Municipal Election

Em

18 "t--Em

m

City of Anywhere

Em

Em

~

20

~

~

~

FIG. 3

~

~

~

~

~

m

~

~

17

u.s. Patent

Juo.5,2012

US 8,195,505 B2

Sheet 5 of7

~

~

m

~

~

~

m m m m m

tz?ZI tz?ZI

January 11891

tz?ZI To vote, completely fill in the box(es)

tz?ZI tz?ZI

BALLOT FOR THE OFFICE

~

VoteforonlyONE (1) candidate

,,~

1) RobertL.

tz?ZI

3) Jacques

tz?ZI

4) Samuelde

tz?ZI

5) James

tz?ZI

6) Leif

IZ?Zl tz:Zl

tz:Zl

10)William L.M.

tz?ZI

11)Wilfrid

tz?ZI

13)LesterB.

CHAMPLAIN COOK

HUDSON

KING LAURIER MacDONALD PEARSON

tz:Zl

14)Louis S.

SAINT-LAURENT

tz?ZI

15)PierreE.

TRUDEAU

tz?ZI

I

Nick Salvator


"T1

.",

'<

'<

('D'

...,-l


ml

-I G')

:;0

;;0

G')

f223 ml

ml

mI

0 ."

ml

."

ml

VI

;:0; "T1

ml

ml

::::> 0.

-g

ml

-l

;;0 ;;0

Iii

ml ml ml ml

mI ~

m

~

m m m

~

m

~

ORO: 2109 Office of Mayor: 0,0,0,1,0,0,0,0,1,0,0,0,0,1,0,780 Vote for: Nick Salvator . Office of Councillor: (800,0), (1,790), (0,0) Vote for: Miles J. MASSEY (1st), Menelaus P. O'OANIEL (2nd) Office of ALDERMAN: Vote for: OVER-VOTE

ml

"C

tz:Zl

~

ml

ml

JOLIET

12)John A.

ml

MASSEY .0 0 1) MilesJ. O'DANIEL O. 0 2) MenelausP. VON ESPIE 00 0 3) Clause ~ 0 $ .!:!! ..:e ~ ~ o ...,:J:l> .-- 2: <- m :;;: CD mr c: 0' 0 ~~ -<'" n ~ '"'":::s:::s -<~. Q ...,0 c: ~ riC; 0 m;;u 0 :J: 0 ;;u m m 0 ~ :;;: 0 0 z 0: » s:: 0 0men n n m r 0 n s: 2: 0 rn » s: "TI 0 ,m m 0 m C z » 2: ::;0 n 0 m 0 -< 0 S » z 0 Ii iii liD 1110 :;0

tz:Zl

ml

CARTIER

8) Henry 9) Jake

ml

ml

FRANKLIN

tz?ZI

Candidate 1 DO Candidate 2 • 0 Candidate 3 O.

ml

,r} "\,~~

ERIKSSON

7) John

COUNCILLOR

ml

CABOT

-2}_Jean

~

BALLOT FOR THE OFFICE OF

Votefor ONE (1) 1Stchoice & ONE (1) 2ndchoice,as shown at right

BORDON

•

o next to your choice(s), like this:

MAYOR

OF

tz?ZI

18

!p-si

General Municipal Election

tz?ZI

ml

ORO

City of Anywhere

~

2,812,2,823,788,774

)

FIG.4

( 30

m

'( 17

u.s. Patent

Jun.5,2012

Sheet 6 of?

US 8,195,505 B2

Data Card from Data Card from Data Card from Tabulator 1 Tabulator 3 Tabulator X Data Card from Data Card from Tabulator 2 Tabulator 4

Computer Central Election Database

32

I

( 28

Auditing PC Auditing PC Auditing PC Auditing PC Station #1 Station #2 Station #3 Station #Y

FIG. 5

I

u.s. Patent ~ ~

~

~

~

To vote, completely fill in the box(es) BALLOT FOR THE OFFICE OF

MAYOR

~

Voteforonly ONE (1) candidate

fZZZJ

1) Robert L.

~

2) Jean

~

3) Jacques

~

4) Samuel de

~

5) James

~

6) leif

fZZZJ

~ ~ t'Z2Zl ~ fZZZJ

~

~

~

~

~

~

CABOT CARTIER CHAMPLAIN COOK ERIKSSON

7) John

FRANKLIN

8) Henry

HUDSON

9) Jake

JOLIET

10) William LM.

KING

11)Wilfrid 12)JohnA. 13) lester B.

~

14) louis S.

~

15) Pierre E.

t'Z2Zl

lAURIER MacDONALD PEARSON SAINT·lAURENT TRUDEAU

NiCK Salvator

0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

~

~

~

•

o next to your choice(s), like this: BALLOT fOR THE OFFICE OF

COUNCILLOR Candidate1 DO Candidate2 • 0 Candidate3 O.

shown at right

,~

'\,~~

""Tl

.£!

s:

~

r <= n

'" -<'" "'":;:: => :::>

'"<;

~ ~ tzZ3 ~ ~

0 0

~

CD

'"=> ct>

......

<0-i0l o "T1:Z: l> (i) m,..... 0'-

m

:2. -<

Q

~S

0"

m;;o

no

t:: '0

-i

~ ~ ~ l?Zl

:z:

I.?Za

m m

l?Zl

::0

~ n '"~0.

s:

0

;;0

0 en

-0 (")

Z --i Cl

0

m

z

;;>;;

m

0

0

0

""Tl

r-

""Tl ;;0

;s: m

:::c -<

(")

;s:

z

(")

rc: Cl

» r

is:

;s:

n

tz23

»

z

m

m '"

«)

(")

»

:::c :::c

rIl~IIiD~D

ea

m

C

l?Zl

;::0

:s:

~

» z

I2Zl

IIW

l?Zl I2Zl

~

m

~

~

~

~

m

~

~

~

ORO: 2109 Office of Mayor: 0,0,0,1,0,0,0,0,1,0,0,0,0,1,0,780 Vote for: Pierre E. TRUDeau Office of Councillor: (800,0), (1,790), (0,0) Vote for: Miles J. MASSEY (1 st), Menelaus P. O'DANIEL Office of ALDERMAN: Vote for: OVER-VOTE

~

VON ESPIE

~ ~

~

O'DANIEl

~ ~ z '~ => iii'

~

~

MASSEY

2) Menelaus P.

~

~

liD DPII

1) Miles J.

3) Clause

~

ij·si

Votefor ONE (1) tst choice & ONE(1) 2ndchoice, as

BORDON

~

ORO

General Municipal Election January 11891

~

~

~

City of Anywhere

~

~

US 8,195,505 B2

Sheet 7 of7

Juo.5,2012

(2nd)

2,812,2,823,788,726

FIG.6

(

30

\17

US 8,195,505 B2 1

2

SYSTEM, METHOD AND COMPUTER PROGRAM FOR VOTE TABULATION WITH AN ELECTRONIC AUDIT TRAIL FIELD OF INVENTION This invention relates generally to technologies for tabulating votes. This invention relates more particularly to ballot boxes with vote tabulation capabilities. The invention also relates to method for tabulating votes in an election.

1

0

BACKGROUND OF THE INVENTION Numerous voting technologies are known. These technologies are generally directed to permitting votes to be recorded with efficacy while maintaining the secrecy of the ballot. One category of voting technology involves the use of the paper ballot. Various methods exist for collecting and tabulating votes recorded by paper ballots. Prior art has been created to rely on paper ballots while using modern digital technology. For example, "Optical-mark Based Systems" allow voters to mark paper ballots and feed the ballots through an automated optical ballot scanning tabulator into sealed ballot boxes. The optical scanning technology determines the nature of the vote and the computer technology provides a means to quickly tabulate results. The results are transmitted to a central computer by a variety of transmission technologies. All results are tallied at the central computer. Another category of voting technology is best described as "Mechanical Voting Systems". These systems generally utilize mechanical counters for Counting votes. These systems include lever machines and punch card ballots. Another category of voting technology is the "Direct Recording Electronic" (DRE) system. This type of system permits voters to enter their vote into a digital system by means of an electronic interface such as a touch screen, mouse or scrolling cursor. Generally speaking, an interface device in each voting station is linked to a computer system. The results compiled by each set of voting stations are then transmitted to a central computer system using a variety of computer architectures and transmission modes. In a variation of the DRE system, the entry of a vote into a digital system is linked to a paper receipt that is used as a voter verified paper receipt. This paper receipt can be used in a paper-based audit procedure-an operation that is impossible in a true DRE system. There is a need for voting technologies that provide the ability to audit the election results. Democracy depends on accurate and incorruptible tabulation of voter intent; so by extension, voting technologies must be able to demonstrate vote tabulation integrity. One of the challenges in permitting efficientverification of voting results, and also demonstration of the integrity of the vote, are the subjective elements generally involved in determining voter intent. In accordance with numerous prior art voting technologies, the assessment of what constitutes a vote in favour of a particular candidate or a particular answer to a question is determined by subjective assessment. This is best illustrated in the context of a manual hand count. Historically,voters would select their candidates using pen and paper, making a mark (an 'X' for example) beside the candidate(s) of their choice. This paper ballot would then be manually counted by election officials with candidates and their agents allowed to scrutinize the operation. However,by the very nature of elections, different parties involved with elections have very different interests. A set of candidates and

15

20

25

30

35

40

45

50

55

60

their agents rarely share common motivation and accordingly the interpretation process of what constitutes a mark on a ballot has always been prone to inconsistencies. This fact generally makes it difficult to achieve the same tally when counting the same ballots multiple times, as there is no assurance that any given ballot would be interpreted the same way by every interested party. The reason for this is straightforward: often the voter would make a mark that could be deemed as ambiguous by anyone of the people involved with the manual ballot count (i.e. election official or candidate agent). What may be obvious to one agent may not be obvious to another agent. In addition, many ballots were deemed 'spoiled' or 'blank' because the voter (either intentionally or unintentionally) made too many or too few marks on the ballot. The apparent lack of objectivity was addressed in a number of prior art systems. The Mechanical Based Systems and Optical-Mark Based Systems either incremented a particular vote or not based on objective standards. The Mechanical Based Systems have been deemed susceptible to inconsistencies in well publicized cases. For example, some mechanical punch systems have been shown to leave cut-out 'chads' partially attached, leading to disputes surrounding election results. Also, in the case of Mechanical Based Systems and Optical-Mark Based Systems, the objective standards for what constitutes an indication of voter intention are generally set by the manufacturer of the voting device and have not been adaptable to conform to objective standards determined by election officials for a particular vote. For example, in Optical-Mark Based Systems an integration threshold is usually defined to establish how dark a mark has to be in order to constitute a mark recognized by the system as being such. This integration threshold is permanently set by the manufacturer. These concerns have hindered the ability to verify vote results, and have generally had a negative impact on impressions of vote integrity. To some extent these problems have been addressed by DRE systems. Voter intention in a DRE system is a binary input from a digital device so ambiguity caunot exist. However, it is well known that processing errors, computer memory corruption or computer memory loss can occur in digital devices and there have been well publicized reports suggesting that this may be the case with some prior art DRE systems. If this type of failure were to occur,voter intention is permanently corrupted or lost and a new election would have to be held. In addition to the aforesaid disadvantages, the prior art solutions generally do not enable an election officialto effectively audit voting results and/or verify the integrity of the vote. For example, the abovementioned prior art technologies do not permit an election official to pick a ballot at random and examine the results of the optical scan algorithms that determined voter intent. This ability to compare the paper ballot with the results collected by a vote system allows the election officialto verify the accuracy of each voting unit and ensure that there are no random or systematic tabulation errors in the voting system or subsystems. There is a need for a voting system and ballot collection apparatus that enables objective measurement of voter intention and a method to verify accurate data collection to demonstrate the integrity of the vote. SUMMARY OF INVENTION

65

A tamper proof and secure ballot box includes a slot that permits paper ballots (which include one or more voter selec-

US 8,195,505 B2 3

4

tion entry areas to be manually completed by a voter using a suitable marking device) to be fed into the box by means of a paper feed mechanism. The interior of the box includes a ballot processing device. The ballot processing device includes a computer and a digital scanning device. The ballot processing application of the present invention is loaded on the computer. The digital scanning device scans each paper ballot fed into the box to create a digital image thereof, in a manner that is known. The ballot processing application analyses each digital image based on predetermined criteria. The predetermined analysis criteria include co-ordinates of marking locations, co-ordinates of ballot identifiers, locations of relevant scripts, and pixel-based threshold values for determining voter intent (marked, unmarked, ambiguous). The ballot box includes an alarm device that signals incorrect ballots or when a ballot has been determined to require resolution of intent by the voter (erroneous or ambiguously marked voter selection entries). In this case, the ballot is returned by the feeding mechanism, following which normal procedures are followed to cancel the ballot and issue a new ballot to the voter. Another aspect of the inventionis that the ballot processing application creates a "vote stamp" based on analysis of each digital image. This vote stamp is attached to a digital image of the scanned ballot and stored as a single data file to a memory linked to the computer. The vote stamp may include a randomly assigned unique identifier corresponding to a particular digital image of a ballot. The vote stamp also may include data corresponding to the results of the analysis of the digital image conducted by the ballot processing application. Another aspect of the vote stamp is that it may include other data relevant to the election process such as the polling station where the particular ballot was scarmed. The paper ballot is retained within the ballot box of the present invention in a secure marmer and then handled in accordance with the specificrules of the election in which the ballot box is used. The various data files corresponding to the digital ballot images and corresponding vote stamp are stored in the memory in random order. The voting selection of a particular individual cannot be discerned by individuals observing the order in which ballots are fed into the ballot box. The computer also includes a suitable back-Lip storage device and related utilities to reduce the risk of data loss from the memory. The system of the invention includes a central processing computer. The memory linked to the computer consists of a computer cartridge. Data from a plurality of computer cartridges from a plurality of ballot boxes is retrieved and stored to a central memory linked to the central processing computer. One aspect of the present inventionis that the association of the vote stamp with the corresponding digital image enables rapid and cost effective verification of the integrity of the processes. For example, a random selection of a defined percentage of ballots processed in accordance with the invention can be examined for accurate collection of voter intent by calling up from the central memory the digital images and corresponding vote stamps. (for example from ballots received in a particular polling station) and by comparing the digital image displayed on a computer screen with the results of the processing thereof as shown by the data included in the vote stamp. Another aspect of the present invention is that it provides the objectivity and speed of a digital process in connection with the use of paper ballots in an electoral process. A written

record of the electoral choice in the form of the paper ballot is maintained as well as a cost savings as fully electronic voting (for example using touch screens) is generally more costly. One object of the present invention is to provide a system and method that enables an electoral process that can be audited efficiently, accurately and transparently. Another object of the present invention is to include the aforesaid auditing capabilities in a machine that can be easily accessed by any persons with visual, motor or language disabilities.

10

BRIEF DESCRIPTION OF THE DRAWINGS

15

20

25

30

35

A detailed description of several embodiments of the present inventionis provided herein below by way of example only and with reference to the following drawings, in which: FIG. 1 is a flowchart illustrating the method of the present invention, in accordance with one particular aspect thereof. FIG. 2 is an illustration of the tabulation unit of the present invention, in a cross-section thereof. FIG. 3 is an illustration of a paper ballot that provides an example of a paper ballot processed in accordance with the present invention. FIG. 4 illustrates a scanned and processed image of the paper ballot illustrated in FIG. 3 that includes the vote stamp footer of the present invention. FIG. 5 is a diagram illustrating a system architecture representative of one embodiment of the system of the present invention. FIG. 6 is a further scanned image of another paper ballot that includes a vote stamp footer, wherein the vote stamp footer reflects processing results inconsistent with the voter intention. In the drawings, preferred embodiments of the invention are illustrated by way of example. It is to be expressly understood that the description and drawings are only for the purpose of illustration and as an aid to understanding, and are not intended as a definition of the limits of the invention. DETAILED DESCRIPTION OF THE INVENTION

40

45

50

55

60

65

One aspect of the present invention is a method that includes the following steps: 1. Establishing one or more criteria for analyzing a plurality of paper ballots, the paper ballots including one or more voter selection areas, which criteria include threshold values for determining whether a particular voter selection area has been filled in, has not been filled in, or it is ambiguous whether or not it has been filled in; 2. Scarmingat least one of the paper ballots using a scanning device, the scanning device generating a digital image document of the each paper ballot; 3. Analyzing the digital image document to determine whether a particular voter selection area has been filled in, has not been filled in, or is ambiguous as to whether or not it has been filled in, so as to define a series of processing results for each digital image; 4. Creation of a vote stamp which includes the processing results and attaching the vote stamp to the digital image document; and 5. Comparing the processing results with the digital image document corresponding to a paper ballot, thereby enabling the election results to be audited on a per ballot basis. A further aspect of the method of the present invention is that the threshold values are based on pixel Count. The threshold values are established by election officials, as particularized below.

US 8,195,505 B2 5

6

In another aspect of the invention, the vote stamp includes other data relevant to the election process such as the polling station where the particular ballot was scanned. Further aspects of the method of the present invention are explained below. Overall, the method of the present invention is illustrated in FIG. 1. FIG. 2 illustrates one particular embodiment of the invention which consists of a hybrid paper/electronic vote tabulator 10. The hybrid paper/electronic vote tabulator or tabulation unit 10 includes a lock downtype ballot box 12, earphones 14, and a pendant manual trigger 16. As voters come into the polling location, they are generally processed by the election official who determines voter eligibility (based on local election rules), and also determines the proper ballot for the voter. Based on the voter's requirements, they are either issued a paper ballot, or issued an 'audio' ballot, in accordance with one aspect of the present invention. FIG. 3 illustrates a representative paper ballot 17, which includes a plurality of voter selection areas 18. In the case of the particular paper ballot 17 illustrated, there is also an election officialarea 20, which appears in most ballots, and is used by the election official to validate the particular ballot. Usually the validation takes the form of a signature in the election official area 20 by the election official, however, other means of authentication can also be used. The election official area 20 serves to mark the paper ballot 17 as having been officially issued. It should be understood that the paper ballots 17 can either consist of pre-printed ballots or a ballots printed locally ondemand by means of a regular laser printer. The voter will generally take the validated blank paper ballot 17 to a private area (such as a desk with privacy barriers), and make a mark or marks (this can be done in numerous ways such as filling in a box, making an 'X', etc) beside the chosen candidate(s) with a permanent marker. For the sake of clarity, this disclosure generally refers to selection of a candidate or candidates, however, some votes relate to other questions such as referendum questions and the like. The present invention extends to votes obtained for a voter mandate on all matters whether candidates, weighted candidate votes, referendum questions and the like. The voter then generally takes the marked paper ballot 17 to the tabulation unit 10, where it is fed through the hybrid tabulator 10, as shown in FIG. 2. Secrecy of voter intent is preferably maintained by using a confidentiality folder made from paper (not shown). The tabulation unit 10 includes a ballot processing device 22. The ballot processing device 22 includes a computer (not shown) and a ballot reader or digital scanning device 24. The ballot processing device 22 also includes a known paper feed mechanism 26. The ballot 17 is fed through the paper feed mechanism 26 and the digital scanning device 24 takes a high resolution optical scan of the ballot 17. In one particular aspect of the invention, the tabulation unit 10 takes the resulting high resolution scauned image (not shown) of the entire ballot 17, and saves one copy of this image to non-volatile memory (not shown) linked to the computer (not shown) so as to create a permanent record thereof. Loaded on the computer is the ballot processing application 28 which constitutes the computer program aspect of the present invention. The ballot processing application 28 randomly assigns a filename to each scanned image (not shown) to ensure that the order in which the ballots 17 were scauned remains private. The ballot processing application 28 also sends another copy of the images (not shown)to the computer (not shown) for image recognition.

In a particular aspect of the invention, the computer (not shown) is linked to the digital scanning device 24, and the ballot processing application 26 defines an image recognition routine (not shown) that is applied to the digital scanning device 24. One aspect of the image recognition routine (not shown)is that the ballot processing application 28 enables the computer to selectively recognize specific areas of each image (not shown) and analyze such specificareas as particularized below to define a series of processing results (not shown) associated with the particular ballot 17. For example, the ballot processing application 28 enables the computer to analyze the various security markings on the ballot 17 to ensure it is a valid ballot, perform a pixel count check to verify that the election official area 20 was initialled by the election official to ensure the ballot was properly issued to an eligible voter, and perform a pixel count of each voter selection area 18 on the ballot 17. In addition, as further examples of such processing results (not shown), depending on the pixel count of each marking box, the mark is either classified as a 'vote', a 'non-vote', or an 'ambiguous marking'. These classifications are based on the total pixel counts of the marking areas ("marking area" or "marking areas" refer for example to voter selection areas 18 or election official areas 20 and other similar areas) of the ballot, and the corresponding pixel levels defined by the election officials at a given time prior to the election. The election officials clearly define, in pixels, the minimum pixel count that is to be classified as a 'vote', the maximum pixel count (if any) that is to be defined as a definite 'non-vote', and a range of pixels in between those values that will constitute an 'ambiguous marking'. These pixel values are loaded on each tabulation unit 10.After calculating these pixel values for each marking area, the tabulation unit 10 will return the ballot (by operation of the paper feed mechanism 26) to the voter with an appropriate error message for any of the following scenarios (as a particular illustration of the present invention): any marking area was classified as an 'ambigious marking'; too many marking areas in one category were classified as 'votes' resulting in an "over-vote"; no marking areas in one or more category were classifiedas 'non-votes', resulting in a blank ballot; and/or no pixel count was recorded in the election official area 20, indicating that the ballot was not initialed by an election worker. It should be understood that the tabulation unit 10 can be programmed to allow a voter to verify its ballot in the case of over-voted or blank ballots, thus preserving the voter's right to cast an over-voted or blank ballot. If the ballot was not returned to the voter for any of the above reasons, the computer (not shown) determines that all of the votes pertaining to the particular ballot 17 have been cast by that ballot, and appropriately increments a main counter (not shown) that forms part of the memory (not shown), in a particular embodiment of the present invention. In addition, once the determination of the total votes for that ballot 17 has been made, by further operation of the ballot processing application 28, the computer (not shown) appends a footer 30 to the saved ballot image (not shown), which contains all of the processing results (not shown) for that specific ballot 17, as shown in FIG. 4. The footer 30 can also be understood as a "fingerprint" or a "vote stamp" that is attached to each image (not shown). The present invention thereby permits the creation of one random, clear, image of every ballot 17 along with all the information the tabulation unit 10 has compiled for that particular ballot 17 that can be later viewed by any software image viewer (not shown).

10

15

20

25

30

35

40

45

50

55

60

65

US 8,195,505 B2 7

8

Upon successful completion of the ballot processing, in accordance with the present invention, the ballot 17 is passed the rest of the way through the tabulation unit 10, where it drops into the ballot box 12, where it mixes freely with the other ballots to preserve the anonymous ordering of the ballots (as shown in FIG. 2). In a particular aspect of the present invention, means is provided for voting by an individual with a challenge such as visual impairment, illiteracy, inability to read the language in which the ballot is written, physically impairment with only limited motor abilities, etc. In such case, in accordance with another aspect of the method of the present invention, the election official initializes an 'audio ballot' (not shown) which is administered directly by the tabulation unit 10 (in one particular embodiment thereof). By operation of another aspect of the ballot processing application 28 of the present invention, the election official initializes, or activates, the audio ballot by using an electronic key (or some other suitable mechanism), and ushers the voter to a voting area where the voter can put on the ear phones 14 and take hold of the handheld pendant that includes the manual trigger selector 16. The audio ballot (not shown) is best understood as an audio version of the ballot 17, which is stored to the memory (not shown) associated with the computer (not shown). The voter negotiates through the ballot 17 through the audio interface and makes candidate selections via the manual trigger 16. Everytime the voter makes a selection, and every time a voter completes voting for every individual race, the computer (not shown) asks the voter to confirm the choices after it is audibly reviewed. Upon completion of the audio ballot, a final review is read back to the voter, and the voter is asked one more time to confirm the selected choices. This audio clip of this review, along with the subsequent sound of the voter confirming the review, is recorded and randomly stored in memory (not shown) by operation of the ballot processing application 28. In addition, once the voter is finished with this final review, the election official gives the voter one last chance to cancel the ballot and start over. Once this option has been denied by the voter, the election official uses their electronic key to cast the audio ballot. By doing so, the votes from that audio ballot are appended to the ongoing tally of the tabulation unit 10 (as particularized above), the sound bites are saved to the memory, and optionally a printer linked to the computer (not shown) prints the votes summary of the audio vote onto a paper chit. The chit then falls into the ballot box 12 in order to maintain secrecy or is printed in encrypted form for audio playback on either the tabulation unit 10 or external device. This aspect of the method of the present invention is also illustrated in FIG. 1. At the end of voting, the tabulation unit 10 includes an ongoing tally for all the ballots that passed through that particular machine, and those results are transmitted (by various known methods and technologies) to a central location which includes a central election database 32, as illustrated in FIG. 5. The various election results from the various tabulation units 10 are summed together in a mauner that is known and stored to the central election database 32, in a manner that is known. It should be understood that one aspect of the present inventionis a system consisting of the tabulation unit 10 of the present invention, and which includes the computer (not shown) and the ballot processing application 28. Another aspect of the present invention is a further system consisting of a plurality of tabulation units 10 linked to a central computer (not shown), the central computer (not shown) being linked to the central election database 32.

The system and computer program aspects of the present invention can be provided in a manner that is known to those skilled in the art. Particular hardware and software components are described below for illustration purposes. Technical Description of the Tabulator Hardware The tabulation unit 10 is generally a self-contained unit made up of severaldifferent subcomponents. The following is a list and description of representative subcomponents. Motherboard In one particular aspect of the present invention, the ballot processing mechanism 22 includes a motherboard that provides the functional block of the tabulation unit 10, and which controls electronic processing of the tabulator process. The motherboard (not shown) preferably includes several interfaces, including an interface with the paper feed mechanism 26, an audio/general purpose input output (GPIO) interface, a scanning card and image heads (associated with the digital scanning device 24), the power supply board, a thermal printer, and the user interface 34 illustrated in FIG. 2. The motherboard (not shown) includes the computer (not shown) consisting of a known microprocessor. The motherboard (not shown)along with its processor (not shown)can be made to operate with a variety of software operating systems, but currently is utilizing either WINDOWS™ CE.Net or uClinux™. The processor (not shown) may consist of an x86 INTEL™ based processor (for the Windows CE.Net version) or, for example, a MOTOROLA COLDFIRE™ processor (for the uClinux™ version). The motherboard (not shown) has various sources of memory including, for example, onboard flash memory where the operating system kernel resides, as well as various hard drive options including an on-board ATA flash drive and a removable compact flash drive. Paper Feed Mechanism The paper feed mechanism 26 is that part of the tabulation unit 10 that physically controls the movement of the paper ballot through the tabulation unit 10. In a particular embodiment of the present invention,the paper feed mechanism 26 is of a very similar in construction to todays modern sheet-fed scanning systems (or sheet-fed fax machines) in that it incorporates a system of rubberized wheels connected to a stepper motor that are responsible for advancing the paper either forward or backward through the paper feed mechanism. The paper feed mechanism 26 also houses a object sensor system which is a series of LED/photo-detectors used to detect a sheet of paper that has been inserted into the opening slot of the paper feed mechanism 26.Also included on the paper feed mechanism 26 is a double paper sensor, which is a commercially available module that is capable of detecting more than one sheet of paper. Lastly, the paper feed mechanism 26 also houses two separately packaged contact image sensor (CIS) units (one abovethe paper feed plane, and one below allowing for duplex imaging), which are commercially available and in large scaleuse in commercial fax and scanningmachines. The paper feed mechanism 26 is preferably designed in such a way as to allow the rubber wheels to advance and reverse the paper through the paper feed mechanism and across the mounted CIS units in such a manner as to allow no more than 7 degrees of paper skew,and in such a manner as to ensure that the paper does not bind with any part of the tabulator, thus ensuring that the paper does not jam inside the tabulator. The opening slot 36 of the paper feed mechanism 26 is extended out through a front slot 38 of the ballot box 12 for the insertion of the paper, and a rear slot is 40 extended through the rear wall of the paper feed mechanism ensuring that the paper is able to freely exit the paper feed mechanism and fall directly into the ballot box 12.

10

15

20

25

30

35

40

45

50

55

60

65

US 8,195,505 B2 9

10

Scanning Card and Image Heads The digital scanning device 24 of the present invention preferably includes a series of imaging heads or modules (not shown), which can be either be of the contact image sensor (CIS) or charge coupled device (CCD) type. When prompted, each imaging module (either CIS or CCD) takes a one dimensional image of the ballot paper that is in direct contact with the module, producing a series of black and white or greyscale pixels that represent the image for that particular line of the paper. By coupling this process with a single line advance of the stepper motor (i.e. advancing the paper by one line) and repeating, a very high quality two dimensional image of the entire sheet of paper is created. The imaging modules interface directly with the image controller. In one form of the tabulation unit 10, this controller is in the form of a separately mounted custom printed circuit board (PCB). This board contains an application specific integrated circuit (ASIC) chip that has been designed for specificuse in scanning applications. This chip is commercially available and is in wide use in commercial scauningproducts. Inits other form, this imaging controller resides directly on the Coldfireprocessor which is located on the main motherboard. In this version of the controller, the software that is encoded on the ASIC chip is ported directly onto the main processor, thus eliminating the need for a separate PCB. In either case, the imaging controller is one particular aspect of the ballot processing application 28 of the present invention. The function of this imaging controller is to control the feeding and scanning of each ballot 17. When 'armed' (i.e. the main tabulator application sends the signal to the image controller to be ready to accept and scan ballots) the image controller monitors the paper sensor hardware. When a ballot 17 is inserted into the opening of the tabulator (i.e. the front of the paper feed mechanism system 26), the paper sense hardware trips, and sends the signal to the image controller to start the scan routine. In this scan routine, the image controller repeatedly triggers the imaging modules to scan a line and then advances the paper feeding wheels (i.e. via the stepper motor) and repeats until the paper has been completely scanned. When the paper is completely scanned, two complete images are sent by the imaging controller to the main tabulator application for processing. During this time, it is important to note that the paper has not been completely fed through the paper feed mechanism. Even though the ballot has completely passed over each imaging head, the bottom piece of the ballot is still being held by the rear rubberized wheels. It is at this point where the imaging controller awaits instructions from the main processor or computer (not shown) for its next instruction. This next instruction can be either to completely return the ballot to the voter (i.e. reversing the direction of the stepper motor allowing the rubberized wheels to move the paper back through the machine and out the front slot) or to accept the ballot (i.e. forwarding the stepper motor a predefined number of steps to allow the ballot to continue through past the rear slot 40 of the tabulator, allowing the ballot to fall into the ballot box). After this instruction, the controller will 're-arm' itself in preparation for the next paper ballot until such time as the main tabulator application tells it to tum off. Power Supply The power supply is generally a separate PCB that converts the incoming power voltage (either 120VAC,240 VAC,or 30 VDC) to a DC voltage that is required by the other PCB's in the tabulator. Specifically, -5VDC, +5VDC, +12VDC and -12VDC are the outputs of this PCB, and are used to power the motherboard, the scanning board, the user-interface module, and the thermal printer.

Audio/GPIO Card The audio/GPIO card is either in the form of a separate PCB, or built-in on the motherboard. Its function is to provide an audio output and a GPIO input to the motherboard. The card interfaces to two mini 3.5 mm connectors that are mounted on the rear plate of the tabulator; one is where the earphones 14 plug in, and the other is where the handheld pendant that houses the manual trigger 16 plugs in. This card allows the main application to play various sound filesvia the earphone jack, and accept user input via the handheld pendant. Thermal Printer This is a commercially available thermal printer which interfaces directly with the motherboard, in one particular embodiment of the present invention mentioned above. The printer is mounted in Such a way that the paper output is aligned with a slot in the tabulator cover, allowing the printed piece of paper to feed through the tabulator cover to the outside. When sent a printjob by the motherboard, the printer produces a printout on a thermal paper roll. User Interface The user interface 34 is a known interface device mounted to the tabulator cover, which possesses both an information display screen and a method of user input. In a preferred embodiment of the present invention, there are currently two different types of this module. The first is a TFT display screen coupled to a resistive touch-screen input device. Both are commercially available pieces of hardware. The second type of user-interface module available is a module that possesses an LCD display screen with several keypad buttons. Both modules interface directly with the motherboard, and are controlled by the ballot processing application 28. It is through this device that election officials are able to interface with the tabulator. This interface is used for a number of situations, including entering administrative passwords, setting up audio ballots, final casting of audio ballots, printing various types of election reports, accepting or returning overvoted or blank ballots, ctc., which constitute further functions of the ballot processing application 28. Chassis and Exterior Package The chassis is made up of a metal base and metal rear panel which is mounted to the base at 90 degrees. The power supply PCB, the scanner PCB, the motherboard PCB, thermal printer and audio/GPIO board are all mounted to this metal chassis. The paper feed mechanism is then mounted to the base as well, but sits above the above other modules. The user interface module is fastened to the paper feed mechanism cover (which can be either plastic or metal) which in turn fastens to the metal base and rear plates via tamper-proof security screws (i.e. torx screws). Description of Scanning Software As mentioned above, one aspect of the ballot processing application 28 of the present invention is an application layer that calls the scanning function that points to a matrix. When a ballot 17 is scanned a driver of the ballot processing application 28 is responsible to copy the digital scan of the document to a specified location on the memory (not shown). When the scanner is enabled it is in ready/active mode and will await the insertion of a ballot 17 in the scanner. In this mode of operation, the digital scanning device 24 will await the insertion of a ballot to begin the scan. The operation of the scanner and driver during the scanning process is outlined below: I. When a ballot 17 is inserted into the digital scanning device 24, the front Object Sensor will send a signal to the scanning board which will notify the driver. If the driver is in active mode, it will start scanning the document.

10

15

20

25

30

35

40

45

50

55

60

65

US 8,195,505 B2 11

12

2. At the start of each line scan, the driver will send a 'start pulse signal' to the CIS module (where a CIS module is used, as explained above). This prompts the CIS module to place an analog output voltage on its Vout pin that is representative of the darkness of the first pixel. This is followedby a sequential series ofVout voltages for each of the pixels in the line. These signals are spaced by the internal CIS clock, which is setby the driver.As the digital scanning device 24 is going through its line scan, the scanning board will convert each voltage signal to a digital signal, in a mamler that is known. The threshold value of this quantization is configurable. 3. The resultant digitized image (not shown), again, is stored in the predefined location of the memory (not shown). 4. When the back Object Sensor is triggered, the scanning board will notify the driver. At this point the driver will notify the application layer (via a function call) that the document has been scanned, imaged, and is sitting in matrix form in memory. The driver will also, via this function call, let the application know the size of the matrix. Here, the driver will be put in a wait state. It will wait for instructions from the application layer. S. A driver definedAPI is provided so that the application can then instruct the driver to either pass the document completely through (forward) the digital scanning device 24, or completely reverse the direction of the stepper motor, returning the entire document back through the front of the unit. Also if the driver detects any errors in the document (i.e. double sheet, paper jam, etc.) then it will reverse the document and notify the application layer of the error. During this disabled state the driver will disregard the begiuning of document signal from the device and not scan any documents (i.e ballots). The digital scauning device 24 is initially in this inactive state. It will be in this state until the initialization routine is complete and everything is okay. It can be put back in this inactive state by the application layer. The overall speed of a scan from once the paper is sensed to when the driver notifies the application layer that a ballot 17 has been scanned is generally under one (1) second, using existing technology, which is suitable for the purpose of the present invention. The driver described will generally allow the scan of any sized document up to a maximum of 21 inches long. Description of Random Auditing Process As described above, once the vote has been completed, the vote result filesare transferred to the central election database 32. Election officials can now perform a random auditing process, in accordance with another aspect of the invention,to ensure that the tabulation units 10 have functioned correctly (and also to verify the integrity of the vote overall). This can be in the form of a spot check, a structured check to gain a certain confidencerate in the tabulator integrity,or a complete check of every ballot cast. For the sake of clarity, once the vote result files from each tabulation unit 10 have been transferred to the central computer (not shown), there will be one multimedia file for every ballot cast during the election, complete with the corresponding processing results (not shown) or vote stamped 'fingerprint' information. In the case of paper ballots, there is one image file for every ballot cast, and in the case of audio ballots, there will be one sound file for every audio ballot cast. Depending on the auditing strategy used, the auditing committee (for example) can have any number of terminals accessing the ballot files from the central election database 32, as illustrated in FIG. 5. Each auditor working on a separate computer can open as many ballot files as needed.

Audit Process The process that allows auditors to verify that any given ballot was tabulated properly is as follows. As explained above, a complete image of each ballot 17 scanned is savedto the ballot file. In addition, the fingerprint of that ballot (which contains all the pertinent information that the tabulator obtained from that ballot, including pixel values for each marking area as well as which candidates where voted for) is vote stamped below that image by including same in the footer 30. By simply opening any given ballot file on their computer terminal, the election auditor is able to compare what the voter markings to the tabulator interpretation. In the case of the audio ballot, the election auditor can compare the sound filethat the voter reviewed and confirmedwith what the tabulator recorded for that ballot. Thus by simply opening one of these multimedia files, the auditor can verify whether the tabulation unit 10 made an error in tabulating that particular vote. In FIG. 6, we see an example of a ballot that was incorrectly tabulated. It is clear that the voter intended to vote for the write-in candidate 'Nick Salvatore', but the tabulator registered a vote for Pierre E. Trudeau. In essence, to verify that one ballot has been tabulated correctly takes only a few seconds. Types of Auditing Strategies The main advantage of this type of fingerprinting and corresponding audit trail is that it easily enables virtually any auditing strategy. On the one extreme, perhaps only one or two ballot filesfrom each tabulation unit 10would be checked to verify that the tabulator fingerprint matched the original ballot information. This would provide some degree of a sanity check against a systematic problem, but depending on the number of ballots cast at each tabulator, would generate a fairly low confidence rate on the auditing process. On the other extreme, would be a 100% confidencerate on the audit. For this, every single ballot file would be opened to confirm that the tabulator fingerprint matched the original ballot information. However,depending on how many ballots had been cast during the election and how many audit workers were available to carry out the auditing, this process could be too slow. It should be understood that the present invention can be used in many voting scenarios, including for example shareholder votes and the like. Specificaspects of the hardware and software described can be modified without departing from the system and computer program aspects of the present invention. What is claimed is: 1. A method of tabulating votes, comprising the steps of: (a) establishing one or more criteria for analyzing a plurality of paper ballots, each of the plurality of paper ballots including one or more voter selection areas, said one or more criteria including threshold values based on pixel count that indicate at least one of the following determinations: (i) that a specific voter selection area that is one of the one or more voter selection areas has been selected; (ii) that the specific voter selection area has not been selected; or (iii) that it is ambiguous whether or not the specificvoter selection areas has been selected; said threshold values thereby determining voter outcomes when applied to pixel counts associated with the one or more voter selection areas; (b) a user inserting one of the plurality of paper ballots into a ballot processing device that scans said paper ballot using a scanning device, the scanning device generating

10

15

20

25

30

35

40

45

50

55

60

65

US 8,195,505 B2 13

14

a digital image of the paper ballot that is scanned, said 3. The method of claim 1, comprising the further step of the pixel-based analysis involving basing the threshold values on digital image including one or more digital voter selecpixel sizes that determine the voter outcomes. tion areas which represent the one or more voter selec4. The method of claim 1, comprising the further step of tion areas of the paper ballot; returning to the user, who is a voter who cast the paper ballot, (c) a ballot processing application of the ballot processing upon one or more negative ballot outcome determinations device that is linked to the scarming device analyzing the based on the threshold values: that the ballot is incorrect; or digital image to determine a ballot outcome including at that the voter outcome is ambiguous. least one of the following: 5. The method of claim 1, comprising the further step of (i) that one or more particular digital voter selection 10 one or more election officials establishing the threshold valareas that are one or more of the one or more digital ues. voter selection areas have been selected in accordance 6. The method as claimed in claim 1, comprising the further with the pixel-based threshold value that indicates step of further analysing the digital image for one or more of that at least one of the one or more digital voter selecthe following: tion areas is selected; 15 (a) one or more co-ordinates for marking locations; (ii) that the one or more particular digital voter selection (b) one or more co-ordinates for ballot identifiers; and areas have not been selected in accordance with the (c) one or more locations on the ballot for scripts. pixel-based threshold value that indicates that at least 7. The method of claim 1, comprising the further step of one of the one or more digital voter selection areas is storing the digital image and the vote stamp attached to the not selected; or 20 digital image in the memory linked to the computer, or the one (iii) that it is ambiguous whether or not the one or more or more storage devices, said storage being in association particular digital voter selection areas have been with the random identifier, such that no particular digital image and vote stamp attached to the particular digital image selected in accordance with the pixel-based threshold can be associated with a particular voter, said random identivalue that indicates that at least one of the one or more 25 fier maintaining secrecy of voter intent. digital voter selection areas is ambiguous; S. The method of claim 1, comprising the further steps of: and analyzing the digital image to determine additional auditing the election results by randomly selecting one or information to be obtained from the digital image, more digital images, each of the one or more digital images including at least the total votes for the paper ballot having the vote stamp attached; retrieving said one or more corresponding to the digital image; 30 digital images having the vote stamp attached; and checking (d) the scanning device ejecting the paper ballot and the that the vote stamp matches the information on the ballot that ballot processing device either: returning the paper balthe vote stamp is attached to. lot to the user who inserted the paper ballot into the 9. The method of claim 1, comprising the step of utilizing ballot processing device if the ballot outcome is ambiguany software image viewer to view the single data file. ous; or depositing the paper ballot into a ballot box if the 35 10. A system for tabulating votes and providing an audit ballot outcome is not ambiguous, said ballot box being trail comprising: attachable to the scanning device, and said ballot box (a) a ballot box, said ballot box providing secure storage for providing secure storage for the plurality of paper balone or more paper ballots; and lots; (b) a ballot processing device integrated with the ballot (e) the ballot processing application generating a series of 40 box, comprising: processing results for the digital image, said series of (i) a paper feed mechanism that includes an insertion slot processing results being based on the analysis determifor feeding a paper ballot into the ballot processing nation by the ballot processing application, and said device; analysis including a pixel-based analysis, said pixel(ii) a computer; based analysis being a pixel count analysis; 45 (iii) a digital scanning device, linked to the computer, (f) the ballot processing application creating and appendthat scans the paper ballot, and after the paper ballot is ing a vote stamp to the digital image, which vote stamp ejected from the digital scarming device the paper includes at least information that the ballot processing ballot being either: inserted into the ballot box in application obtained from analyzing the digital image, accordance with a pixel-based threshold value analyand further at least includes the series of processing 50 sis of the computer that determines that the ballot is results, other data relevant to the election process includnot ambiguous; or returned to a user who inserted the ing at least the polling station where the paper ballot paper ballot into the ballot processing device, in associated with the digital image is scanned, pixel values accordance with a pixel-based threshold value analyfor each marking area, information pertaining to one or sis of the computer that determines that the ballot is more candidates voted for, and a random identifier cor- 55 ambiguous; and responding to the digital image; and (iv) a ballot processing application loaded on the com(g) the ballot processing application storing the digital puter, said ballot processing application being proimage and the appended vote stamp as a single date file grammed, for each of the one or more paper ballots, to one or more of the following: a memory linked to the to: computer; or to one or more storage devices. 60 (A) provide instructions to the computer to initiate the 2. The method as claimed in claim 1, comprising the further digital scanning device to scan the paper ballot to step of auditing election results by comparing the digital create a digital image of said paper ballot; image that corresponds to one of the plurality of paper ballots (B) process the digital image, said processing includto the series of processing results provided in the vote stamp ing a pixel-based analysis, said pixel-based analyattached to the digital image to achieve a verification of the 65 sis comprising a pixel connt analysis, to establish a integrity of a voting process incorporating the plurality of series of processing results defining one or more ballots. voting outcomes for the paper ballot, said process-

US 8,195,505 B2 15

16

ing results defining an audit trail, and to determine the threshold values, based on pixel count, that indicate that one or more voter selection areas is: selected, not selected, or ambiguous; (C) determine that it is ambiguous whether or not the one or more particular digital voter selection areas have been selected in accordance with the pixelbased threshold value that indicates that at least one of the one or more digital voter selection areas is ambiguous; (D) provide instructions to the computer to cause the digital scanning device to return the paper ballot to the user who inserted the paper ballot into the ballot processing device when it is determined that the ballot is ambiguous, in accordance with the pixelbased threshold value that indicates that at least one of the one or more digital voter selection areas is ambiguous; and (E) when it is determined that the ballot is not ambiguous, in accordance with the pixel-based threshold value that indicates that digital voter selection areas are selected or not selected, create and append a vote stamp to the digital image and store the vote stamp and digital image as a single data file, said vote stamp including at least the series of processing results, the polling station where the paper ballot associated with the digital image is scanned, pixel values for each marking area, information pertaining to one or more candidates voted for, and a random identifier corresponding to the digital image, (F) wherein the ballot processing application stores the digital image and the appended vote stamp as a single data file to one or more of the following: a memory linked to the computer; or to one or more storage devices. 11. The system of claim 10, wherein the system allows one or more election officialsto establish one or more criteria for analyzing each of the one or more paper ballots. 12. The system of claim 10, wherein each of the one or more paper ballots include one or more voter selection areas, and the ballot processing application is programmed to define criteria for analyzing each of the one or more paper ballots, said criteria including the threshold values to determine the following: (a)that a particular voter selection area being one of the one or more voter selection areas has been selected; (b) that the particular voter selection area has not been selected; or (c) that it is ambiguous whether or not the particular voter selection area has been selected; and

saidthreshold values determine voter outcomes when applied to pixel counts associated with the one or more voter selection areas, and the system applies said criteria to determine the voter outcomes. 13. The system of claim 12, wherein the threshold values are based on pixel sizes that determine the voter outcomes. 14. The system of claim 12 wherein the ballot processing device is programmed to return one of the one or more paper ballots to the user who is a voter who cast the paper ballot upon one or more of the following determinations: that the paper ballot is incorrect; or that the voter outcome is ambiguous. 15. The system of claim 12, wherein the system allows one or more election officials to establish the threshold values. 16. The system of claim 10, whereby the system supports an audit of the voter outcomes of the one or more paper ballots by allowing a user of the computer to compare the series of processing results with the single data file including the digital image and vote stamp that corresponds to one of the one or more paper ballots. 17. The system of claim 10, wherein the digital image is further analyzed for one or more of: (a) one or more co-ordinates for marking locations; (b) one or more co-ordinates for ballot identifiers; and (c) one or more locations on the ballot for scripts related to the processing results. 18. The system of claim 10, wherein the digital image and the vote stamp attached to the digital image are stored as a single data file in the memory linked to the computer, or the one or more storage devices, said storage being in association with the random identifier, such that no particular digital image and vote stamp attached to the particular digital image can be associated with a particular voter, said random identifier maintaining secrecy of voter intent. 19. The system of claim 10, wherein the system allows the auditing of the election results by: randomly selecting at least one single data file that includes the digital image and vote stamp; retrieving the at least one single data file from the memory linked to the computer, or the one or more storage devices, or a remote computer associated with the computer; and checking that the vote stamp of the at least one single data file matches the information on the digital image of the ballot that the vote stamp is attached to, and using a screen linked to the computer or the remote computer to display the at least one single data file or using any software image viewer to view the at least one single data file. 20. The system of claim 10, wherein the ballot processing application is programmed to supportthe processing of one or more audio ballots.

10

15

20

25

30

35

40

45

50

* * * * *

111111

(12)

(54)

1111111111111111111111111111111111111111111111111111111111111 US008714450B2

United States Patent

(10)

Arnao et al.

(45)

SYSTEMS AND METHODS FOR TRANSACTIONAL BALLOT PROCESSING, AND BALLOT AUDITING

Patent No.: Date of Patent:

US 8,714,450 B2 May 6, 2014

(58) Field of Classification Search USPC 235/51-57,386; 705/12 See application file for complete search history.

(75) Inventors: Frederico Arnao, Oakland, CA (US); Eric Coomer, Broomfield, CO (US); Larry Korb, Moraga, CA (US); Josh Maletz, Oakland, CA (US)

(56)

References Cited U.S. PATENTDOCUMENTS

2002/0084325 Al 2007/0170253 Al

* *

7/2002 Reardon 7/2007 Chung et al.

235/386 235/386

(73) Assignee: Dominion Voting Systems, Inc., Denver, CO (US)

* cited by examiner

( *)

Primary Examiner - Kristy A Haupt (74) Attorney, Agent, or Firm - Holland & Hart LLP

Notice:

Subject to any disclaimer, the term of this patent is extended or adjusted under 35 U.S.C. 154(b) by 0 days.

(21) Appl. No.: 13/092,600 (22) Filed:

Apr. 22, 2011

(65)

Prior Publication Data US 2012/0061468 Al

Mar. 15,2012

Related U.S. Application Data (63) Continuation of application PCT/US2009/061493, filed on Oct. 21, 2009.

No.

(60) Provisional application No. 611193,062,filed on Oct. 24,2008. (51) Int. Cl. G07C 13/00

(52) U.S. Cl. USPC

(2006.01)

(57)

ABSTRACT

A voting unit and a ballot transaction processing system for processing a voting session. The voting unit includes an optical ballot scanner and a printer that is capable of printing a non-human readable barcode on ballots that are fed into the voting unit. The ballots are filled out (or cast) by the voter before being inserted into the voting unit. The voter can confirm the voting unit's interpretations ofhislher selections on the scanned ballot via a display and confirm that the interpretations are correct via an input section on the voting unit. The ballot transaction processing system is also capable of generating, assigning and communicating to a plurality of voting units a set of unique transaction codes. In so doing, the ballot transaction processing system maintains voter secrecy, prevents ballot stuffingand provides a one-to-one correspondence between images of optically scanned ballots and the ballots themselves.

235/386; 705/12; 235/51

14 Claims, 5 Drawing Sheets

PROCESSING A VOTING SESSION ASSIGNING A UNIQUE TRANSACTION NUMBER

S101

PRINTING TRANSACTION NUMBER

S102

SCANNING THE PRINTED TRANSACTION NUMBER

S103

READING THE PRINTED TRANSACTION NUMBER

S104

VERIFYING THE PRINTED TRANSACTION NUMBER

S105

INCLUDING A TRANSACTION NUMBER WITH ALL RECORDS OF THE BALLOT

S106

TABULATING BALLOTS

S107

IMPORTING TRANSACTION NUMBERS

S108

u.s. Patent

May 6, 2014

US 8,714,450 B2

Sheet 1 of 5

DEMONSTRATION

BALLOT

FamousNames100wI StraightParty INSTRUCTIONS TO VOTERS: To VOTE, CONNECT THE

5

ENOOOI002_~ ----L

2

ARROW pointing to your like this ...... To vote for a write-In candidate, Write the person's name on the blank line and CONNECT THE ARROW: ~

4 6

.. F~:..:::.:;.::..-:.:__-

..

-1------_

..

~~~jVO~~~F~a~rO~M~_::_:j77~~~~~~~rGJf~~~~~~~~--~~~~~~---~ ..

I ZACHARYTAYLOR&

MILLARDFILLMORE

..

\fll(jjnia Panv

ASSOCIATE JUSTICE OF THE SUPREME COURT

"..

vote FDr One

...

..

!~~::~I~T~~~~~~~ & .. OI'l'()P;orty

F===~-"" ..

F=:..:::..::==-1-----_

7-

--l---r- ______

F~-'----

..

F::'=';_:;_--"

..

I-------

..

d~~~~

,:..J.:=~---

...... .... ....

..

•

• FIG. 1

u.s. Patent

May 6, 2014

US 8,714,450 B2

Sheet 2 of 5

11

/

1

FIG.2

u.s. Patent

May 6, 2014

US 8,714,450 B2

Sheet 3 of 5

11 VOTING UNIT DISPLAY

22

TRACKING DEVICE

NETWORK CONNECTORS

34

28 CPU

32 AUDIO DEVICE

.____

SCANNER

29

-

33

INPUT DEVICE

24

FIG.3

~

PRINTER

30

u.s. Patent

May 6, 2014

Sheet 4 of 5

US 8,714,450 B2

BALLOT TRANSACTION PROCESSING SYSTEM

ELECTION MANAGEMENT SYSTEM (EMS) 43

VOTING UNIT

11

PRECINCT TABULATOR 46

FIG.4

41

u.s. Patent

May 6, 2014

Sheet 5 of 5

US 8,714,450 B2

PROCESSING A VOTING SESSION

ASSIGNING A UNIQUE TRANSACTION NUMBER

r--- S101

PRINTING TRANSACTION NUMBER

------- S102

SCANNING THE PRINTED TRANSACTION NUMBER r-- S103

READING THE PRINTED TRANSACTION NUMBER

r--- S104

VERIFYING THE PRINTED TRANSACTION NUMBER r-- S105

INCLUDING A TRANSACTION NUMBER WITH ALL RECORDS OF THE BALLOT

------- S106

TABULATING BALLOTS

r--- S107

IMPORTING TRANSACTION NUMBERS

r-- S108

FIG.5

US 8,714,450 B2 1

2

SYSTEMS AND METHODS FOR TRANSACTIONAL BALLOT PROCESSING, AND BALLOT AUDITING This application is a continuation of international application PCT/US09/61493, filed Oct. 21, 2009, which claims the benefit of U.S. Provisional Application No. 611193,062filed Oct. 24, 2008. The disclosures of which are hereby incorporated herein by reference in their entirety. BACKGROUND In the technology of ballot transaction processing, it is desirable to develop apparatus and methods for processing paper ballots, such as by optically scanning or optically reading those ballots, in a more efficient and reliable manner. The improvements described herein relate to technologies for processing ballots, tabulating votes and in particular technologies for auditing all tally results from an optical scan-based ballot tabulator. It is generally known to print certain identifying information on a ballot when the ballot is printed before an election. For example, U.S. Pat. No. 6,892,944 discloses providing on each ballot a voter registration number that can include a barcode, two-dimensional barcode, a prescribed font, optical character recognition characters, alphanumeric characters, non-alphanumeric characters and symbols. Further, this patent discloses that the voter registration number can include information such as the voter's state, county, precinct etc. in addition to a randomly generated number that is printed on the ballot prior to election. However,this pre-printed information violates voter secrecy and does not prevent ballot stuffing or provide an adequate one-to-one correspondence between images of optically scanned ballots and the ballots themselves. There are numerous voting technologies known that are directed to permitting votes to be cast and recorded efficiently while maintaining the secrecy of the ballot. Of paramount importance in an election process is the efficientverification and auditing of voting results. One of the obstacles associated with the verification is that there is usually a subjective determination made when determining the voter intent. For example, in the case of mechanical based systems that punch out a hole in a ballot, subjective determinations have had to be made in well-publicized cases to determine the voter intent with respect to partially attached chads. These subjective determinations lead to inconsistent results and have a negative impact on public perception of vote integrity. Therefore, it is desirable to provide a ballot processing system that objectively determines voter intent in a consistent and reliable manner and that provides a mechanism for auditing the results on a vote-by-vote basis. Of equal importance is a process of tracking the transaction of a ballot while the vote is being reviewed prior to casting. To this end, various systems have been developed to track the voting process. However,these systems can be susceptible to tampering. Therefore, it is desirable to develop a more efficient and secure means of tracking a ballot transaction. While many systems provide for auditing vote totals and recounting results, it is desirable to provide a more secure, efficient audit process that can audit an individual vote across all records of this vote including the paper ballot, the electronic record of the cast vote, and the digital image of the ballot.

10

15

20

25

30

35

40

45

50

55

60

SUMMARY 65

In view of the above issues, the following improvements are presented.

One improvement relates to a voting unit including an optical ballot scanner and a printer that is capable of printing a non-human readable barcode as well as human readable characters on ballots that are fed into the voting unit. The ballots will have been filled out (or cast) by the voter before being inserted into the voting unit, and thus before having the barcode and/or human readable characters printed on it by the printer. The barcode and/or human readable text is only printed on the ballot after the voter has cast the ballot (that is, after the voter has accepted the voting unit's interpretation of the voter's selections made on the ballot). The improvement also relates to an election preparation system capable of generating, assigning and communicating to a voting unit a set of unique, randomly generated transaction codes. Some benefits of this improvement are as follows. The election preparation system pre-allocates a set of unique transaction numbers to be used by each voting unit. Ballot processing is done via session transactions where ballots consisting of multiple sheets are logically associated with each other.This allows the system to accurately determine the number of ballots cast, and the true voter turnout independently of the number of ballot cards processed. This system prevents ballot stuffing by ensuring that only ballot image records with valid ballot image identifiers matching those pre-assigned to the associated precinct/voting unit are tallied. Therefore, 'Stuffed' ballots that do not match the pre-allocated set of unique transaction numbers will be rejected and flagged for investigation. This system allows the entire lifecycle of a cast ballot from time of casting to tabulation to be monitored and audited. This system also allows the captured images of ballots to be used for post processing and auditing and to be accurately correlated with the associated physical ballot where necessary, in a verifiable manner. In addition, according to some embodiments, the proposed method does not assign the ballot identification number to a ballot until after the ballot has been cast, thus preserving the secrecy of the ballot. Finally, the application of the ballot identification number effectively "cancels" the cast ballot thus preventing the same ballot from being counted more than once. The ballot transaction processing system described herein includes, for example, a voting unit having an optical ballot scanner, and an election preparation system configured to generate, assign and communicate to the voting unit a set of randomly generated transaction codes. Each voting session (each time a voter inputs the voter's ballot) that is processed by the voting unit can be assigned a ballot sheet identifierthat includes at least a number identifying the particular ballot and a unique transaction number. The unique transaction number can be randomly selected from a pre-assigned list of numbers assigned to that voting unit for that election by the election preparation system. The voting unit preferably also includes a printer. The printer prints one or both of a non-human readable code and human readable characters on ballots that are fed into the voting unit after the ballot has been read by the voting unit and then accepted by a voter. The printer can print a non-human readable code on the ballot representing the assigned ballot sheet identifier.Further, a verification portion can be provided in the voting unit that can read and verify that the assigned ballot sheet identifier has been correctly printed onto every ballot fed into the voting unit. The voting unit includes a display that displays information related to the selections marked on the ballot by the voter after the optical ballot scanner scans the ballot. The voting unit also includes an input section configured to allow the voter to accept or reject the selections marked on the ballot based on the information displayed on the display.

US 8,714,450 B2 3

4

In another aspect of the ballot transaction processing system, the pre-assigned list of numbers can be encrypted. In a further aspect of the ballot transaction processing system, if the ballot is read by the voting unit but is then rejected by the voter (for example, because the voter disagrees with the voting unit's interpretation of the voter's ballot selections), the printer can print a non-human readable code on the ballot representing the assigned ballot sheet identifier plus a code indicating that the ballot has been rejected and human readable indicia (such as the word "canceled") indicating that the ballot was rejected. In yet another aspect of the ballot transaction processing system, the voting unit can read and verify that a printed barcode representing an assigned ballot sheet identifier and rejection condition is correctly printed onto the ballot. In an additional aspect of the ballot transaction processing system, all scanned images, and electronic "Cast Vote Records" of the ballot can be associated with the assigned ballot sheet identifier, thereby providing a one-to-one correlation between the ballot images and the physical ballots. In a further aspect of the ballot transaction processing system, ballots already having an assigned ballot sheet identifier cannot be inserted by a voter and re-scanned and tabulated by the optical ballot scanner. In an additional aspect of the ballot transaction processing system, the ballot transaction processing system can further include a tabulation system that only accepts ballot image records that have ballot image identifier codes that match the codes pre-assigned to an associated precinct/voting nnit by the election preparation system. Another aspect includes a method of processing a voting session of a voter. The method includes: scanning a ballot after a voter has marked his/her selections on the ballot; displaying information related to the selections marked on the ballot by the voter after scanning the ballot; prompting the voter to either accept or reject the selections marked on the ballot based on the displayed information; and printing a unique code on a ballot having election-choice-information printed thereon after the ballot has been cast and accepted by the voter. Additionally, after printing the unique code on the ballot, the ballot can be rescanned to produce an image of the ballot with the unique code printed thereon. The method of processing a voting session of a voter can also include generating three separate records of each ballot including the ballot having the nnique code printed thereon, the image of the ballot and the unique code, and results information related to the ballot including the unique code. Yet another aspect includes a method of processing a voting session of a voter, the method including: assigning to a cast ballot a unique transaction number that is randomly selected from a list of numbers assigned to a voting unit for an election; printing a non-human readable code such as a barcode on the ballot after the ballot has been read by the voting unit and after the read ballot has been accepted by the voter (that is, after the voter has accepted the voting unit's interpretation of the ballot); scanning the printed barcode on the ballot; reading the printed barcode on the ballot; verifying that the printed barcode has been correctly printed on the ballot; associating the unique transaction number with all of the records associated with the cast ballot, including in the digital image of the ballot, and in an electronic record of the ballot; tabulating the ballot and other ballots that have valid transaction numbers and importing the transaction numbers into an election management system.

apparent from the following descriptions of exemplary embodiments with reference to the accompanying drawings, in which like numerals are used to represent like elements and wherein: FIG. 1 is a diagram illustrating an example of a ballot; FIG. 2 is a diagram illustrating an example of a voting nnit; FIG. 3 is a diagram illustrating some of the components of a voting unit; FIG. 4 is a diagram illustrating some of the components of a ballot transaction processing system; and FIG. 5 is a flowchart illustrating some of the steps of a method for processing a voting session of a voter.

10

DETAILED DESCRIPTION 15

Ballot Transaction Processing

20

25

30

35

40

45

50

55

60

BRIEF DESCRIPTION OF THE DRAWINGS 65

The foregoing and further objects, features and advantages of the apparatus and methods described herein will become

FIG. 1 illustrates an example of a ballot 1. The ballot 1 can be, for example, 4.25 inches or 8.5 inches wide and from 11 inches to 22 inches in length. In one embodiment illustrated in FIG. 1, the ballot 1 has ballot registration marks 3 which are solid black 0.25 inch squares locatedjust inside of a 0.25 inch unprinted area, bounding all sides of the ballot 1. Where the ballot 1 is longer than 11 inches, additional registration marks are desirable and can be provided. FIG. 2 illustrates an example of a voting unit 11 that can be an optical ballot scan device.As seen from FIG. 2, voting unit 11 can include an input slot 23 into which a ballot 1 to be scanned is fed, a ballot feed tray 38, a display 22, an audio device 33, and a user-manipulatable input device 24. FIG. 3 illustrates some of the components that can be included in each voting unit 11. The voting unit 11 can include a CPU 32 that controls operation of the unit 11 including the functions described herein, a tracking device 34, an audio device 33, an input device 24, an optical scarmer29, a printer 30, network connectors 28 and a visual display unit 22. Voting unit 11 is not limited to these specific components as any number of other components known to one of ordinary skill in the art for inclusion on voting nnits could be incorporated therein. After a voter fills-ina ballot, the voter inserts the completed ballot into the slot 23 of the voting unit 11. The voting unit 11 then optically scans the ballot with its internal scarmer 29, which can be a CCD scanner, for example. An image of the scanned ballot then can appear on the display 22. By viewing the image, the voter can confirm that the ballot image is correct. In addition, by using image recognition/processing technology (see, for example, U.S. Pat. No. 6,854,644, the disclosure of which is incorporated herein by reference in its entirety), the voting unit 11 determines the selections made by the voter on the ballot (i.e., determines which candidates, etc. were selected by the voter) and displays those determined selections to the voter via display 22. The user can then confirm that the voting unit's determinations are correct, and if correct, accept the voting unit's determinations via input device 24. FIG. 4 illustrates some of the features that can be included in a ballot transaction processing system 41. The ballot transaction processing system 41 can include, for example, an Election Management System (EMS) 43, a voting nnit 11 and a precinct tabulator 46. The precinct tabulator 46 mayor may not be provided in the voting nnit 11. Each voting unit 11 is a tabulator in that each voting unit 11 consolidates the votes cast on the voting unit 11 at the end of the night to create a machine results report. In addition, there may be a secondary precinct tabulator/consolidation unit that consolidates and tabulates the results from "several" voting units 11. Further, the EMS also acts as a

US 8,714,450 B2 5

6

"central" tabulation and accumulation system consolidating all of the results from all of the precinct tabulators 46 in the election. FIG. 5 is a flowchart illustrating a method of processing a voting session of a voter. Each voting session (that is, a voter's set of selections for the ballot 1) processed by the voting unit 11 is assigned a unique transaction number (S101 of FIG. 5) that is randomly selected from a pre-assigned list (set) of numbers assigned to that voting unit 11 for that election after the voter accepts the voting unit's determinations regarding the ballot. The preassigned list of numbers may be encrypted to prevent spoofing. When a ballot 1 has been read (optically scanned) by the scanner 29 of the voting unit 11 and then accepted by the voter, the printer 30 of the voting unit 11 prints (S102 of FIG. 5) a non-human readable barcode 2 (for example) on the ballot 1 representing the assigned transaction number (see FIG. 1). The optical ballot scanner 29 of the voting unit then again scans, reads and verifies (S103, S104, S105 of FIG. 5) that the printed barcode 2 has been correctly printed onto the ballot 1. The scanned portion of the ballot with the barcode 2 is appended to the previously scanned (pristine) image of the ballot 1. The electronic cast vote record recorded by the tabulator associated with this ballot is also assigned this transaction code. The electronic cast vote record is a file indicating the voter's candidate (and/or other) selections as interpreted by the voting unit 11 and accepted by the voter. Assigning/printing (S106 of FIG. 5) the same transaction number (also called "transaction code") to all stored records provides a mechanism for tying all 3 independent records of the ballot 1 directly to one another including the physical ballot 1 (including the printed transaction code 2 on the ballot 1), digital image of the ballot (scanned ballot 1 with the printed transaction code 2), and the electronic record (assigned transaction code). To ensure that each ballot 1 is tabulated only one time, the optical ballot scan device will not re-scan and tabulate ballots that already have a transaction code printed on the ballot 1 when the ballot was inserted into the slot 23. This provides one mechanism for preventing 'ballot stuffing'. Further, the tabulation system 46 that tabulates (S107 of FIG. 5) the ballots 1 only accepts ballot results (ballot image records) that have ballot image identifier codes that match the ballot image identifier codes pre-assigned to the associated precinct/voting unit 11 by the election preparation system. This provides another mechanism for preventing 'ballot stuffing'. Finally, all of the assigned transaction codes can be imported (S108 of FIG. 5) into the Election Management System (EMS) (describe below) after the election day tallies have been completed. During audits and recounts of the results, it is possible to ensure that every ballot that was cast and assigned a transaction code on election day is counted during the audit or recount. Any discrepancies during a recount due to ballots 1 being altered, or due to marginal marks can be identifiedon a ballot-by-ballot and vote-by-vote basis by comparing the tallied results from election day and during the recount using the transaction codes assigned to the ballot. In addition, the transaction code prevents the same ballot 1 from being counted more than once during recounts and audits as well. Election Management System The programming of each voting unit 11 can be accomplished through the use of the EMS 43. When the EMS 43 creates an election definition for a voting unit 11 or creates a mechanism for programming a voting unit 11 (whether through a cartridge or through programming directly over a

network), the EMS 43 generates a set of voting transaction numbers to be associated with that voting unit 11 such that the total number of allocated voting transaction numbers will exceed the number of predicted registered voters for that precinct. The transaction numbers can consist of, for example, an election identifier, a precinct identifier (or an early vote identifier) and an additional number. There are a number of approaches that can be taken to allocating these additional numbers. This set of assigned numbers will be unique across the entire election, and can even be unique across multiple elections andjurisdictions ifjurisdiction specific identifiers are included. One approach is that at the time of election definition, the EMS 43 will create a set of these numbers considerably larger than the predicted number of registered voters in the jurisdiction. These numbers could form a continuous set or be randomly generated. The EMS 43 allocates these numbers randomly to the set of allocated session numbers for a voting unit 11 during the election/cartridge creation process for that voting unit 11.Another approach is for the EMS 43 to generate a set of random numbers to be used during the election/cartridge creation process for that voting unit 11, where the size of the set is larger than the predicted number of registered voters for the precinct to which the unit is allocated. Regardless of which approach is used, the EMS 43 assigns a set of unique transaction numbers to each voting unit 11, and provides the voting unit 11with that set along with its election definition. When a voter initiates a voting session on a voting unit 11 (for example, by inserting a completed ballot 1 into the slot 23 of the voting unit 11), the voting unit 11 randomly selects one of the transaction numbers that have been pre-allocated to identify the voting session transactions. Each number will only be allocated once, ensuring that each transaction number allocated is unique to that voting session. If the voting unit 11 includes an optical ballot scanner 29 and the ballot 1 consists of more than one sheet, then each sheet will be assigned the same ballot transaction code (transaction number). This allows the system to determine the true voter turnout regardless of the number of sheets that a given voter casts (sometimes voters do not complete all sheets). The voting unit 11 also includes a printer 30 disposed therein, which is capable of printing a barcode onto the ballot sheet 1. The optical ballot scanner 29 of the voting unit 11 is also capable of reading and decoding barcodes, including barcodes that the voting unit 11 has printed onto a ballot sheet

10

15

20

25

30

35

40

45

1. 50

55

60

65

When the voting unit 11 processes a ballot sheet 1 (and after the voter accepts the voting unit's determination of the voter's selections), the printer 30 prints the transaction number, in the form of a barcode, onto the ballot sheet 1. This ballot imprinter (the printed information) is also capable of identifying counterfeit, fraudulent, or duplicated ballots as well by printing "Invalid" in the same fashion as the barcode may be printed on the ballot 1.After the printer 30 has printed the transaction code, the ballot scanner 29 rescans the ballot 1 to include the barcode in the ballot image and to verify that it has been printed correctly.When the voting unit 11 processes the ballot 1 and saves the votes detected and the images of the ballot 1, it will also save the transaction number assigned to the ballot 1. If a voter inserts a ballot 1 which already has a barcode printed on it representing a transaction number into the slot 23 of the voting unit 11, the voting unit 11 will not process and tabulate that ballot sheet 1 because the bar-coded ballot sheet should have already been processed by that or another voting

US 8,714,450 B2 7

8

unit 11. This prevents a ballot 1 from being read and processed (and counted) multiple times. When the polls have been closed and the results cartridge of a particular voting unit 11 is read into the tally system (precinct tabulator 46), the tally system checks each ballot image record and compares the transaction number to the list of transaction numbers assigned to that particular voting unit 11, to ensure that the voting unit 11 has correctly assigned transaction numbers to all of the records, and no records exist with incorrectly assigned transaction numbers. If a transaction number does not match the numbers assigned to that particular voting unit 11, the tally system will check the number against all the assigned transaction numbers. If there is no match, then the tally system does not tally that ballot image record and stores the ballot image record in a list (file or database) indicating that the ballot was not a valid ballot and not processed on a valid voting unit 11. If the transaction number is valid, but not for the voting unit 11 from which it was read, the tally system will not tally it and will store it in a list indicating that there is a possible issue, which can then be resolved later. The tally system can be a central tally/accumulation system, which is part of the EMS. The tally system is also able to count the number of different transaction numbers that have been tallied. This accurately represents the number of voters that have cast ballots, irrespective of whether they cast all the ballot sheets when a ballot consists of multiple sheets. The tally system also is able to accurately associate the captured digital image of the ballot 1with the ballot record data including the transaction number. This feature assists in post-election auditing and ensures that the digital images can be verified against the physical ballots as well as the electronic cast vote records. During an audit or recount, the recount results can be compared to the results from election day on a vote-by-vote basis utilizing the transaction codes assigned to each ballot. The recount machines can be configured to only accept ballots with transaction codes printed on them that match the transaction codes assigned to voting sessions on election day. After the recount is complete, any discrepancies can be tied back to a single voting session by comparing, on a vote-byvote basis, the cast vote records from election day to the recount results. In addition, the system can be configured to allow each transaction code to be processed once during the recount, and at any time, the system can report any "missing" transaction codes from the recount that were assigned to ballots on election day. Transaction Barcode/Ballot Imprinting System The voting unit 11 employs a ballot imprinting device (printer 30) to both "cancel" and uniquely identify ballot cards after the cards have been processed and "cast" by the voter. The ballot imprinter can be, for example, an inkjet printer with a Ij8 inch wide print head. The barcode can then be printed along a 1/4 inch margin along the side of the ballot card. The main function of this system is to imprint a barcode 2 on the ballot 1 once the voter has verified, reviewed, and accepted the processed ballot results. The barcode 2 includes jurisdictional level identifying characteristics, as well as a randomly generated portion to uniquely identify each voter session. In addition, additional human readable characters can be printed along with the barcode. These characters will aid in manual separation of "exception" ballots; "exception" ballots are ballots that have been marked as having some type of error or rejection criteria. Human readable text can also be added to identify ballots that are counterfeit, duplicated, or otherwise fraudulent in nature.

The barcode 2 can use, for example, Code 128C encoding and contain, for example, a maximum of 10 codewords for a total of 20 numerical digits to form a transaction code. The format of the transaction code can be as follows: MM-YY-ST-CNTY-N,NNN,NNN,NNN MM: 2 digit Month YY: 2 digit Year ST: 2 digit FIPS "State" Code CNTY: 3 digit FIPS "County" Code+1 extra digit N: unique "Voter Session" number with maximum value 9,999,999,999 For example, in this embodiment, for an election conducted in San Francisco County, California in November, the jurisdictional barcode portion would be: 1108060075. The Voter Session number serves a multitude of security and data related purposes. When the election is initiated, all number values are available from 1-9,999,999,999.Each voting device has a pool of these number values randomly assigned to the voting device at the time of initialization. The size of the pool will be dependent on the number of machines used in the jurisdiction. When a voting session is opened (voter introduces a ballot to the voting unit 11), a particular number value from among the pool will be assigned to the electronic record created for the processed ballot results. In addition, this value will be appended to the jurisdictional information described above and encoded into a barcode 2 that is imprinted on the ballot 1once the ballot 1has been cast. It is then possible to tie each electronic record and scanned image directly to the physical ballot 1in the ballot box. In addition, the voting unit 11 can be configured to automatically reject any ballot 1that has already been imprinted with this identifying barcode. This prevents "stuffing" the ballot box as each ballot 1can only be tabulated once because the ballot imprinting acts like a cancellation mark. In addition, ifthere are multiple cards for a single ballot 1, the identifier is associated with the individual card records together at an individual voter level (i.e., the same identifieris printed on each card of a voter's multi-card ballot). This enables the tracking of true voter turnout, which is not possible with current paper-based scanning systems with multiple ballot cards.As these values are assigned randomly from a pre-assigned pool of values, there is no way to tie the identifier to a specific voter, thus ensuring complete voter privacy. Finally, since a pool of identifying values are pre-assigned to each voting unit 11 and stored in encrypted files, the processed data can be compared after tally to ensure that all records have valid identifiersthus further improving the security and audibility of the results. At any time, the image scauned by the voting unit 11 and processed for the initial results can be compared to the physical ballot 1 cast by matching the identifier associated with the electronic record to the barcode 2 imprinted on the ballot 1. The foregoing description is considered as illustrative only of the principles of the improvements discussed above. The inventions described herein are not limited to specific examples provided herein. What is claimed is: 1. A method of processing a voting session of a voter, the method comprising: assigning to a voter-marked paper cast ballot a unique transaction code that is randomly selected from a set of codes assigned to a voting unit for an election; printing the unique transaction code on the voter-marked paper cast ballot after the cast ballot has been read by the voting unit and after the voting unit's interpretation of the cast ballot has been accepted by the voter;

10

15

20

25

30

35

40

45

50

55

60

65

US 8,714,450 B2 9

10

scanning the unique transaction code printed on the cast ballot; verifying that the printed unique transaction code has been correctly printed on the cast ballot; including the unique transaction code with all records associated with the cast ballot; tabulating the cast ballot and other cast ballots that have valid transaction codes; and importing the transaction codes into an election management system. 2. The method of claim 1, wherein the unique transaction code that is randomly selected from the set of codes assigned to the voting unit is encrypted. 3. The method of claim 1, wherein the printing the unique transaction code on the voter-marked paper cast ballot comprises printing a barcode on the voter-marked paper cast ballot. 4. The method of claim 1, wherein the unique transaction code printed on the voter-marked paper cast ballot is used only once for the voting session. 5. The method of claim 1, wherein the set of codes assigned to the voting unit are generated such that the total number of allocated transaction codes exceed the number of predicted registered voters for a particular precinct. 6. The method of claim 1, wherein the transaction code includes an election identifier,a precinct identifier and at least one additional number. 7. The method of claim 1, further comprising: identifying counterfeit, fraudulent, or duplicate ballots; and marking the identified ballots as invalid. 8. A voting unit apparatus comprising: a processor configured to assign to a voter-marked paper cast ballot a unique transaction code that is randomly selected fro.ma set of codes assigned to the voting unit for an election;

a printer coupled to the processor and configured to print the unique transaction code on the voter-marked paper cas~ ballo.t after the cast ballot has been read by the votmg umt and after the voting unit's interpretation of the cast ballot has been accepted by the voter; a scanner coupled to the processor and configured to scan the unique transaction code printed on the cast ballot; the processor further configured to: verify that the printed unique transaction code has been correctly printed on the cast ballot; save the unique transaction code with all records associated with the cast ballot; tabulate the cast ballot and other cast ballots that have valid transaction codes; and import the transaction codes into an election management system. 9. The apparatus of claim 8, wherein the unique transaction code that is randomly selected from the set of codes assigned to the voting unit are encrypted. 10. The apparatus of claim 8, wherein the printing the unique transaction code on the voter-marked paper cast ballot comprises printing a barcode. 11. The apparatus of claim 8, wherein the unique transaction code printed on the voter-marked paper cast ballot is used only once for the voting session. 12. The apparatus of claim 8, wherein the set of codes assigned to the voting unit are generated such that the total number of allocated transaction codes exceed the number of predicted registered voters for particular precinct. 13. The apparatus of claim 8, wherein the transaction code includes an election identifier,a precinct identifierand at least one additional number. 14. The apparatus of claim 8, wherein the processor is further configured to identify counterfeit fraudulent or duplicate ballots and mark the identified bal'lotsas invalid.

10

15

20

25

30

35

* * * * *

111111

(12)

1111111111111111111111111111111111111111111111111111111111111 US008864026B2

United States Patent

(10)

Coomer et al.

(45)

(54) BALLOT IMAGE PROCESSING SYSTEM AND METHOD FOR VOTING MACHINES (75) Inventors: Eric Coomer, Broomfield, CO (US); Larry Korb, Moraga, CA (US); Brian Glenn Lierman, Exeter, CA (US); Doug Harold Weinel, Evergreen, CO (US)

Patent No.: Date of Patent:

US 8,864,026 B2 Oct. 21, 2014

USPC .......................................................... 235/386 (58) Field of Classification Search USPC 235/386 See application file for complete search history. (56)

References Cited U.S. PATENTDOCUMENTS

(73) Assignee: Dominion Voting Systems, Inc., Denver, CO (US) ( *)

Notice:

Subject to any disclaimer, the term of this patent is extended or adjusted under 35 U.S.C. 154(b) by 690 days.

(21) Appl. No.: 13/092,606 (22) Filed:

May 10, 2012

(57)

Related U.S. Application Data (63) Continuation of application PCT/US2009/062078, filed on Oct. 26, 2009.

No.

(60) Provisional application No. 611193,062,filed on Oct. 24,2008. (51) Int. Cl. G07C 13/00

(52) U.S. Cl. CPC

235/386 235/386 235/386

Primary Examiner - Daniel Hess (74) Attorney, Agent, or Firm - Holland & Hart LLP

Prior Publication Data US 2012/0111940Al

235/386

* cited by examiner

Apr. 22, 2011

(65)

6,854,644 BI 2/2005 Bolton 7,077,313 B2 7/2006 Chung 2005/0247783Al * 1112005 Pouloset al. 2006/0202031Al 9/2006 Chung 2006/0266829Al * 1112006 Boltonet al. 2007/0170253Al * 7/2007 Chunget al. 2008/0308634Al * 12/2008 Boltonet al.

(2006.01) G07C 13/00 (2013.01)

ABSTRACT

A ballot processing system and method processes paper ballots, such as by optically scanning or optically reading those ballots. The ballot image processing system corrects for, or is able to differentiate valid voting marks from, ballot printing errors such as skewed printing, incorrect sizing, and speckling. Further, the ballot image processing system, after determining whether each of the voting marks is valid or not, associates audit data with the ballot that corresponds to the decision regarding each voting mark. 27 Claims, 5 Drawing Sheets

DEMONSTRATION BALLOT Famous Names 100 wlStraight Party INSTRUCTIONS TO VOTERS: To VOTE, CONNECTTHE ARROW ooiintina i~ vour choke, liKe this:..... To vote for a writeAin candidate, write the person's

u.s. Patent

Oct. 21, 2014

US 8,864,026 B2

Sheet 1 of 5

- "~5

3

I

-~

----

1

..

• FIG. 1

- ,.

3

.. FIG.2

u.s. Patent

Oct. 21, 2014

US 8,864,026 B2

Sheet 2 of 5

11

/

1

FIG. 3

u.s. Patent

Oct. 21, 2014

US 8,864,026 B2

Sheet 3 of 5

11 VOTING UNIT DISPLAY

22 TRACKING DEVICE 34

NETWORK CONNECTORS

2a CPU

J2 AUDIO DEVICE 33

f--

L....-

2a

I---

INPUT DEVICE 24

FIG.4

SCANNER

'----

PRINTER

3.Q

u.s. Patent

Oct. 21, 2014

US 8,864,026 B2

Sheet 4 of 5

DEMONSTRATION BALLOT Famous Names 100 w/Straight Party

EN000002-

4

INSTRUCTIONS TO VOTERS: ToVOTE, CONNECT THE ARROW pOintingto your choice, like tt)is:~ To vote for a write-in candidate,write the person's nameon the blank line provide and CONNECT THE ARROW' ~ 6 NONPARTISAN STRAIGHT PARTY FEDERAL OFFICES STATE OFFICES OFFICES

us REPRESENTATIVE T~~;;;~~~;t~:~i~ ~~rr{Jr;i\~iy~~i~!~;ic~"e Vote for One 1

VIRGINIA PARTY

$l1li

OHIO PARTY

$l1li

V!lglr1 a Party

CALIFORNIA PARTY

$l1li

OhiO Party

NEW YORK PARTY

$111

W,C. REDFIELD

WILLIAM B. WILSON

LEROY'SATCHEL' PAIGE $ HAROLD 'RED:GRANGE $ D. ROCKEFELLER. IIIiI JOHNNYWEISSMULLER • .C ~O~N ua:wma Perry KNUTEROCKNE • $l1li ~~I $ l1liWilliAM DEMPSEY $ GEORGEBABE RUTH $l1li Write-In .l1li MiLDREDZAHARiAS Ohio Party

Cai:fo:nia Palty

Write-In

Virainia Parri

BENJAMiN HARRISON ADLAIE,STEVENSON

$l1li

&.

CHESTER HRTHUR & • THOMAS A. HENDRICKS THEODOREROOSEVELT& CHARLESW. FAIRBANKS ..

Virginia P8rty

III

NewYork P3:~i

7

~

.l1li

Ohio Party

JOHN HANCOCK California Party

NELSONW,ALORICH

$l1li

.C .l1li

Ohio Party

JOHN MARSHALL

.l1li

.l1li

FRANCIS SCOTT KEY

Cai,fornia Party

.C JOHNJAY New '!ork Pa,ty

$1IIiI

WILLIAM R. HEARST

$l1li

$l1li

Wrile·ln

.1IIiI

Ohio Palty

Caiifoinia Paity

Write-In

NONPARTISAN OFFiCES

MEMBER OF STATE LEGISLATURE 3rd DISTRICT

BOARD OF EDUCATION Vote for One

\/oteforOne

SUSAN B. ANTHONY

Virgin:a Party

MAMIE EISENHOWER

New York Pa:ty

.l1li

Oh:o Party

Write·ln

.111

Ca:,(O'n18.Pa'1"j

•

III CLARENCE DARROW

DIRECTOR OF ENTERTAiNMENT

ANDREW CARNEGIE

~~e'livork Party

U.S. SENATOR \iote to~One EVERETT DIRKSEN .l1li Virginia P8rty CHARLES CURTIS

.1IIiI

\j:rgir;:a Party

III FLORENCENIGHTINGALE.

Caiifornia Party

Write-In

LEARNED HAND

Vote for One

OlliD Partv

$l1li

BOOKERT. WASHINGTON$

$l1li

ALBERT EINSTEIN

ELEANOR ROOSEVELT •

l1li

$1IIiI

IIIiI

Vote for Three

.l1li $111 $1IIiI .1IIiI .l1li .l1li

Write-In

$111

Write-In

.l1li

Write-In

.111

DIRECTOR OF TRANSPORTATION

\/oterorOne IIIiI HENRYFORD

HELENKELLER

$

IIIiI RANSOME. OLDS

DOLLYMADISON fJewVorkParty

.l1li

JOHNDEWEY

$1IIiI

Write·ln

.l1li

Write-In

$l1li

••

CARLOELOMBARD GEORGEE. JESSEL BiLLYROSE KATESMiTH ISADORADUNCAN EDWARDELliNGTON

THOMAS ALVA EDISON $

FIG.5

III III

.l1li

Write-In

Vote fDrOne

STATE SENATOR 37th DiSTRICT

III

C

•• .l1li

ASSOCIATE JUSTICE OF THE SUPREME COURT

STATE OFFICES

III III

$l1li

~o~f~q~~

~JewVork Party

Vote fOI' One

Vote for Two

Virg:nia Palty

ROBERT LA FOLLETTE $ III J. PAUL GETTY

PRESIDENT AND VICE PRESiDENT

DIRECTOR OF RECREATION

Vote for One

CORNELIUS V.A.NDERBIL~ l1li

$l1li

FEDERAL OFFICES JAMESWADSWORTH

ZACHARY TAYLOR & MILLARD FILLMORE

STATE TREASURER

Write-In

.l1li $111 .l1li

•

u.s. Patent

•

Oct. 21, 2014

US 8,864,026 B2

Sheet 5 of 5

II NONPARTISAN JUDICIAL OFFICES JUDGE OF THE SUPERIOR COURT 2nd DISTRICT Vote for One

WilLIAM A.DOUGLAS .II1II JOHN BASSETT MOORE. II1II EARL WARREN .II1II .II1II JOHNSOLOMO~ 7 -~

.II1II

JUDGE OF THE SUPERIOR COURT 3rd DISTRICT

PROPOSITIONS

Do you favor closing all polling places simultaneously 1. throughl the nation during presidential elections?

YES.

.II1II .II1II

Write·ln

JUDGE OF THE SUPERIOR COURT 4th DISTRICT \jute for One

WM. McADOO JOHN ARMSTRONG THOMASTOOD

.II1II .II1II .II1II

Write·ln

.II1II

JUDGE OF THE SUPERIOR COURT 5th DISTRICT VDtefor One

SOJOURNER TRUTH MARY K. MORAN JAMES KENT

.II1II .II1II .II1II

Write·ln

.II1II

NO.

II1II

Are you in favor of the Federal ReserveAct of 1813 2. which places the control of all money in the Treasury

Department?

YES_

II1II

NO.

III(

Shall there be a watervvayjoining the Atlantic and 3. Pacific Oceans througi11heIsthmus of Panama?

YES_

Vote for One

OliVER WENDELLHOLMES. II1II .II1II GEORGE BAER CARRIE NATION .II1II ROGERTANEY

II1II

II1II

NO.

III(

Are you in favor of purchasing tile territory of Louisiana 4. from France?

YES.

II1II

NO.

III(

Are you in favor of turning all gold over to the Treasury 5, Department (1933)? YES. II1II NO. l1li Shall there be free navigation on the Mississippi River 6, (1933)') YES. II1II NO" l1li Are you in favor of abolishing the internal Revenue 7, Service and collecting all taxes through state and iocal agencies? YES. II1II NO. III Shall the President of the United States be elected for 8. no more than three terms?

YES.

II1II

NO. \II1II I

6

II 3

FIG.6

US 8,864,026 B2 1

2

BALLOT IMAGE PROCESSING SYSTEM AND METHOD FOR VOTING MACHINES

ing system is capable of being tolerant of certain printing errors and removes the need for reprinting of ballots in these circumstances. In addition to being able to accommodate the above-mentioned printing errors, this improvement also allows the ballot image processing system to correct for, and distinguish between, spurious marks on the ballot caused by such things as dirt on the ballot, ink smears, dirt smears and the bleed through of ink marks from the opposite side of the ballot. By correcting these spurious marks, the ballot image processing system can reduce the potential misreading of ballots and the need for subjective manual interpretation. This improvement also allows the ballot image processing system to more accurately identify vote target areas and more accurately identify voting intent by accurately identifying actual vote target areas and voting mark types. Therefore, this improvement allows improved and enhanced review of the ballot image for auditing purposes. Additionally, this ballot image processing system allows the voter to resolve write-ins themselves at the voting nnit. The system is able to detect the presence of write-in candidates on a voter-filled-out ballot and support easier manual resolution by allowing the resolution system to display the write-ins to the user, allowing the user to input (e.g., type) the write-in and then associate the typed write-in information with the ballot for subsequent processing. The improvement allows the write-in resolution system to associate a resolved candidate name with the write-in and thus associate it with the ballot image and ballot image record for tally. The system allows the voter to verify the write-in selection through, for example, a touch screen display, thereby ensuring that the correct vote is cast. Specifically, one improvement can include a ballot image processing system having an optical ballot scanner that scans ballots to produce an image of each ballot and an image processing portion that processes the image of the ballot with digital image processing techniques by using predetermined marks on the ballot to correct for ballot skew and image size variations. The image processing portion can be configuredto analyze voting target areas on the ballot and make a target decision regarding each mark by identifying specific allowed voting marks on the ballot to assess voter intentions. Further, a printer can be provided to print audit data onto the ballot after the ballot has been cast by the voter. The audit data can include information regarding each target decision made by the image processing portion. The ballot image processing system can correct for at least one of printing defects and variations, misfeeds and scanning errors. The ballot image processing system also can identify voting target area identification shapes and can select marks from the group consisting of square boxes, rectangular boxes, circles, ellipses, and two arrow ends to be joined. The allowed voting marks can be at least one of horizontal lines, diagonal lines, vertical lines, arrows, crosses, ticks, and filled target areas. The audit data can include at least one of mark categorization, voting mark type and threshold measurements determined by the image processing portion. The ballot image processing system can optionally include a vote tallying system to tally voting results that uses target decision data generated by the image processing portion to supplement the ballot image during review.The supplementing of the ballot image can include at least one of color coded highlights indicating target mark categorization and strength of the target decision data.

This application is a continuation of International Application PCTlUS09/62078 filed Oct. 26, 2009, which claims the benefit of U.S. Provisional Application No. 611193,062 filed Oct. 24, 2008. The disclosures of both are incorporated herein by reference in their entirety. BACKGROUND In the technology of ballot transaction processing, it is desirable to develop apparatus and methods for processing paper ballots, such as by optically scanning or optically reading those ballots, in a more efficient and reliable manner. The improvements described herein relate to technologies for processing ballots and in particular technologies for optically scanning ballots. There are numerous voting technologies known that are directed to permitting votes to be cast and recorded efficiently while maintaining the secrecy of the ballot. It is generally known that ballots can be optically scanned to assist in tabulation and aid in the assessment of voter intent. For example, U.S. Pat. No. 6,854,644 issued to Bolton et al. discloses that a voter's intent can be determined by making a digital image of a mark. The image of the mark is then subjected to a discrimination process that makes a determination as to whether the pixel values in the image correspond to a mark indicating a voter's intent to make a selection. The reference discloses that the determination is reached by comparing the pixel value of a bounded region to a control pixel value set. Further, it is generally known to print certain identifying information on a ballot when the ballot is printed before an election. For example, U.S. Pat. No. 6,892,944 discloses providing on each ballot a voter registration number that can include a barcode, two-dimensional barcode, a prescribed font, optical character recognition characters, alphanumeric characters, non-alphanumeric characters and symbols. Further, this patent discloses that the voter registration number can include information such as the voter's state, county, precinct etc. in addition to a randomly generated number that is printed on the ballot prior to election. Of paramount importance in an election process is the efficientverification and auditing of voting results. One of the obstacles associated with the verification is that there is usually a subjective determination made when determining the voter intent. For example, in the case of mechanical based systems that pnnch out a hole in a ballot, subjective determinations have had to be made in well-publicized cases to determine the voter intent with respect to partially attached chads. These subjective determinations lead to inconsistent results and have a negative impact on public perception of vote integrity. Therefore, it is desirable to provide a ballot processing system that objectively determines voter intent in a consistent and reliable manner and that provides a mechanism for auditing the results on a vote-by-vote basis. SUMMARY In view of the above issues, the following improvements are presented. One improvement relates to a ballot image processing system that corrects for, or is able to differentiate from valid marks, ballot printing errors such as skewed printing, incorrect sizing, and speckling. As such, the ballot image process-

10

15

20

25

30

35

40

45

50

55

60

65

US 8,864,026 B2 3

4

The image processing portion can utilize. DIP algorithms to enhance the ballot images to correct for at least one of speckling, dirt, smears and bleed through. The image processing portion can also use digital image processing techniques to analyze predetermined vote target areas for detection of write-in intent. If write-in intent is detected, the image processing portion can isolate the ballot image of the predetermined vote target area and associate a sub-image of the scanned write-in with the ballot image record. The ballot image processing system can further include a display portion and an input portion. If write-in intent is detected, the display portion can provide a user (the voter) with an opportunity to enter the intended write-in via the input portion. Further, the entered write-in input data can be added to the ballot image record and/or the entered write-in input data can printed on the ballot by the printer in at least one of a human readable form and a machine readable form. The ballot image processing system can further include a write-in resolution system that allows the ballot image records and an associated write-in sub-image of the write-in to be viewed by a user through the display portion and verified by the user by typing in the associated name or selecting a registered candidate through the input portion. The ballot image records, and the associated write-in subimage verified by the user, can be associated with the write-in sub-image and thus the ballot image record. The image processing portion can analyze predetermined target areas to detect at least one of poll worker initials or signatures, polling place IDs and precinct IDs. The optical ballot scanner can analyze predetermined target areas to detect the presence of, and decode 1D and 2D bar codes. A further improvement can include a method of processing a ballot. The method can include: optically scanning a ballot to create an image of the ballot; detecting, using digital image processing techniques, whether defects due to printing and/or scanning exist in the image of the ballot; correcting any defects on the image of the ballot; identifying target areas on the image of the ballot; categorizing the target areas; identifying whether a voting mark is present in the target areas; determining whether each particular voting mark is valid or not; and associating decision information for each voting mark on the ballot. The defects can include at least one of printing defects and variations, misfeeds and scanning errors. The target areas are selected from the group including square boxes, rectangular boxes, circles, ellipses, and two arrow ends to be joined. Valid voting marks can be at least one of horizontal lines, diagonal lines, vertical lines, arrows, crosses, ticks, and filled target areas. The decision information can include at least one of mark categorization, voting mark type and threshold measurements. The decision information is printed on the ballot in at least machine readable form and/or can be saved as a file associated with an electronic record of the image of the ballot.

FIG. 1 is a diagram illustrating an example of a front of a blank ballot; FIG. 2 is a diagram illustrating an example of a back of a blank ballot; FIG. 3 is a diagram illustrating an example of a voting nnit; FIG. 4 is a diagram illustrating some of the components of a voting unit; FIG. 5 is a diagram illustrating an example of a front of a ballot with voter choice selection information printed thereon; and FIG. 6 is a diagram illustrating an example of a back of a ballot with voter choice selection information printed thereon.

BRIEF DESCRIPTION OF THE DRAWINGS The foregoing and further objects, features and advantages of the apparatus and methods described herein will become apparent from the following descriptions of exemplary embodiments with reference to the accompanying drawings, in which like numerals are used to represent like elements and wherein:

10

15

20

25

30

35

40

45

50

55

60

65

DETAILED DESCRIPTION FIGS. 1 and 2 illustrate an example of a front and a back of a ballot 1 before any voter selection information is printed thereon. The ballot 1 can be, for example, 4.25 inches or 8.5 inches wide and from 11 inches to 22 inches in length. In one embodiment illustrated in FIG. 1, the ballot 1 has ballot registration marks 3 which are solid black 0.25 inch squares located just inside of a 0.25 inch nnprinted area, bounding all sides of the ballot 1. Where the ballot 1 is longer than 11 inches, additional registration marks are desirable and can be provided. FIG. 3 illustrates an example of a voting unit 11that can be an optical ballot scan device.As seen from FIG. 3, voting unit 11 can include an input slot 23 into which a ballot 1 to be scanned is fed, a ballot feed tray 38, a display 22, an audio device 33, and a user-manipulatable input device 24. FIG. 4 illustrates some of the components that can be included in each voting unit 11.The voting unit 11can include a CPU 32 that controls operation of the voting nnit 11 including the functions described herein, a tracking device 34, an audio device 33, an input device 24, an optical scanner 29, a printer 30, network connectors 28 and a visual display unit 22.Voting unit 11 is not limited to these specific components as any number of other components known to one of ordinary skill in the art for inclusion on voting units could be incorporated therein. Additionally, the voting unit 11 via CPU 32, which can function as an image processing portion, is able to process a digital image of the ballot and is able to use ballot registration marks 3 (see FIG. 1) on the ballot 1 to correct for ballot skew resulting from effects such as, but not limited to, printing defects and variations, misfeeds and scanning errors. There are numerous known third party commercially available Digital Image Processing (DIP) packages that can be employed that are capable of achieving these effects (such as rotation, skew,scaling, mirroring, pinching). Further,the voting nnit 11is able to process the digital image and to use the ballot registration marks 3 on the ballot 1 to correct for image size variations, caused by effects such as, but not limited to, printing defects and variations. FIGS. 5 and 6 illustrate the front and back of a ballot 1 that has voter and candidate information printed thereon (the ballot has not been filled in by the voter). Voter target areas are illustrated, for example, in FIGS. 5 and 6 as partial arrow 6 and candidate write-in area 7. The image processing portion (CPU 32) is capable of narrowing voting target areas by analyzing the expected location and identifying target area identification shapes and marks for greater location accuracy. For example, image recognition software can be employed to locate the voting target artifacts in a given region of a ballot. The region is a bonnding area of the target location and the image recognition system can

US 8,864,026 B2 5

6

pinpoint the boundaries of the target thereby increasing the is detected, the image processing portion will provide the accuracy of the optical mark recognition and interpretation voter with an opportunity to input the intended write-in (e. g., engine. These target areas can include shapes and marks that by typing the name) or select a registered write-in candidate may include but not be limited to square and rectangular from a list via user-manipulatable input device 24 (such as a boxes, circles, ellipses and two lines (to be joined) known as touch screen, keypad, or audio control box) and thus resolve "Arrow ends". For example, FIG. 5 illustrates an embodiment the write-in themselves at the time of voting. Further, if a where the voting marks consist of separated ends of an arrow 'write-in' intent is entered by the voter via the input device 24, 6 that the voter can counect to cast a vote for a particular which is then associated or recorded with the write-in image candidate. and as a result also is associated or recorded with the ballot Further, the image processing portion can analyze voting 10 image, the ballot image record and the contest, the voter intent target areas and identify specific allowed voting marks to will be clear for auditing purposes. assess voter intentions. Existing systems can calculate the The input data entered by the voter via the input device 24 percentage of pixels within the bounded area, which are black can then be printed on the ballot in at least one of a human or gray above a set threshold. The system can then define a mark as any area that has at least a minimum percentage of 15 readable form and a machine readable barcode 2. An example of a machine readable barcode 2 is illustrated in FIG. 5. The pixels as black. See, for example, U.S. Pat. No. 6,854,644, the machine readable barcode 2 in this embodiment is located on disclosure of which is incorporated by reference herein in its the upper right corner of the front side of the ballot 1 and is entirety. Additional image recognition techniques could be contained within a 0.6 inchx2.75 inch area. FIG. 5 shows a applied to determine ifthere is a continuous line of dark pixels from one end of the target to the other. Such marks may 20 human readable version 4 of the machine readable barcode 2 printed below the machine readable barcode 2. include, but not be limited to: horizontal lines; diagonal lines; vertical lines; arrows; crosses; ticks; and filled target areas. In addition, the image processing portion is capable of The image processing portion is capable of associating analyzing special target areas to detect such artifacts as, but audit data with each target decision that the image processing not limited to: poll worker initials or signatures; polling place portion makes. This data may include but not be limited to: 25 ID' s; and precinct ID' s. Further, the image processing portion mark categorization (voting mark, non voting mark); voting is additionally capable of analyzing special target areas to mark type; and threshold measurements. etc. For example, a detect the presence of, and decode various 1D and 2D bar given target area that was analyzed by the voting system may codes. consist of a rectangular bounding box. The coordinates relaWhen the optical ballot scarmer 29 scans a ballot 1, the tive to the upper left registration mark can be included in the 30 optical ballot scarmer 29 scans the entire image of the ballot 1 target analysis record. This provides the pin-point location the and processes the scanned image using known Digital Image recognition system interrogated to determine whether a mark Processing techniques. These processing techniques apply was detected. If using a percentage of dark pixels, this peralgorithms to the scarmed image to detect features, categorize centage value detected also can be stored with the analysis the detected features and make decisions based on the record. Finally, if the recognition system detected that the 35 detected features. mark within the bounded area was a contiguous line, an Before the image processing portion identifies any voting attribute for 'line' can be added to the record. The audit data marks on the ballot, the image processing portion uses Digital may be stored in a file associated with the scanned image of Image Processing algorithms to correct for aligument defithe ballot, which may be displayed on the display 22 if supciencies in the ballot image. For example, the image processplied with the voting unit 11. The file of audit data can be 40 ing portion detects features on the ballot 1 and is able to stored in the ROM of a computer, a hard drive, a removable determine if the ballot image is skewed, misaligned, or the storage device or any other suitable storage medium. incorrect size. The image processing portion can then correct The improvement also includes a system to tally results the ballot image so that it is straight, centered and the correct that uses the target decision data generated by the image size. processing portion in the voting unit 11 to enhance the ballot 45 Additionally, before the image processing portion identiimage during review. This may include, but is not limited to, fies any voting marks on the ballot 1, the image processing color coded highlights indicating the target mark categorizaportion uses Digital Image Processing algorithms to clean up tion and the strength of the decision. Using the coordinate the ballot image: For example, the image processing portion data of the analysis record, the bounding box defining the can remove the effects of speckling, dirt, smears and bleed optical mark target area can be displayed over the digital 50 through. This ensures that such defects (whether through image. If the system determined that a valid voting mark was printing problems, poor handling or other reasons) will not registered, the bounding box can be displayed with a green adversely effect the detection of valid voting marks or lead to border. If no mark was detected, the bounding box can be the detection of invalid voting marks. displayed with a red border. Next, the image processing portion identifies target areas The image processing portion is also capable of running 55 on the ballot 1 and applies Digital Image Processing algoDIP algorithms to enhance images to correct for such effects rithms to look for, detect and categorize expected voting as, but not limited to: speckling; dirt; smears; and 'bleed target markers. through.' There are commercially available third party image Additionally, the image processing portion is capable of processing packages that are capable of such corrections. detecting different shaped markers, such as the bars that form In addition to detecting target areas on the ballot 1, the 60 "arrow" target markers (see for example FIG. 6), square target image processing portion is capable of analyzing special vote markers, rectangular target markers, circular target markers target areas for detection of 'write- in' intent (see write- in area and oval target markers. When the image processing portion 7 of FIGS. 5 and 6). Ifa 'write-in' intent is detected, the image has detected the correct type of target marker for the election processing portion of CPU 32 will isolate the part of the ballot in the general target areas, the image processing portion calimage containing the write-in selection and associate that 65 culates the center and size of those target markers and uses sub-image with that ballot image record, the scanned image that information to adjust the center and size of the target and the associated contest. Additionally, if a 'write-in' intent areas to be examined for voting marks.

US 8,864,026 B2 7

8

The image processing portion also can examine special target areas and apply Digital Image Processing algorithms to identify the presence of such artifacts such as Poll worker initials or signatures; Polling Place ID's; Precinct ID's. The image processing portion examines the adjusted target areas and applies Digital Image Processing algorithms to identify whether there is a voting mark present. If certain predetermined criteria (such as threshold requirements) are met, the image processing portion categorizes the voting mark as valid. Further, the image processing portion uses Digital Image Processing (DIP) algorithms to identify the types of expected voting marks. For example, the image processing portion is able to detect, identify and categorize horizontal lines, diagonal lines, vertical lines, arrows, crosses, ticks, and filled target areas. Thus, the image processing portion can confirm that a target area does actually contain an authorized voting mark and not a spurious mark. Several commercially available third party pattern recognition packages can be employed to characterize these voting marks. When the image processing portion has detected a mark in voting target area, the image processing portion categorizes the mark and associates decision information to the mark. If the mark has been categorized as a valid voting mark, the image processing portion will associate information such as the voting mark type and decision threshold and statistical information used to make the determination. If a mark is detected but is not categorized as a valid voting mark, the image processing portion will associate information such as the mark type, why the mark is not a valid voting mark and the decision threshold and statistical information used to make the determination. The image processing portion also is able to create an associated overlay image that provides a visual representation (such as color code highlights and shapes) of the categorization and decision-making data associated with marks in the voting areas. These images and the associated data can then be used to assist in audit processes. Further, if the image processing portion detects a valid voting mark in a write-in area, the image processing portion will isolate the area allocated to write in information and create a sub-image of that area which the image processing portion will associate with the ballot image and that particular contest on the ballot image. In the situation where a visual interface and a user input device are provided, the image processing portion will allow the user to verify the presence of the write in and to either type in the intended write-in choice or choose it from a selection of valid registered writein choices. If this is done, that write-in information will be associated with the ballot image, and will be added to the rest of the overlay data so it is able to be overlaid on the image. This feature allows the voter to resolve the write in at the time of voting. If the user resolves a write-in and the voting unit 11 via printer 30 has the capability of printing a bar code 2 and words on the ballot 1, the printer 30 will print the resolved write in information on the ballot 1 and include the write-in information in a printed barcode 2. The foregoing description is considered as illustrative only of the principles of the improvements discussed above. The inventions described herein are not limited to specific examples provided herein. What is claimed is: 1. A ballot image processing system comprising: an optical ballot scauner that scans ballots to produce an image of each ballot; an image processing portion that processes the image of the ballot with digital image processing techniques by using marks on the ballot to correct for ballot skew and image size variations, the image processing portion being con-

figured to analyze voting target areas on the ballot and make a target decision regarding each mark by identifying specific allowed voting marks on the ballot to assess voter intentions; a printer to print audit data onto the ballot after the ballot has been cast, the audit data including information regarding each target decision made by the image processing portion; and a vote tallying system to tally voting results that uses target decision data generated by the image processing portion to supplement the ballot image during review, the supplementing of the ballot image includes at least one color coded highlight indicating target mark categorization and strength of the target decision data. 2. The ballot image processing system of claim 1, wherein the image processing portion corrects for at least one of printing defects and variations, misfeeds and scauning errors. 3. The ballot image processing system of claim 1, wherein voting target area identification shapes and marks are selected from the group consisting of square boxes, rectangular boxes, circles, ellipses, and two arrow ends to be joined. 4. The ballot image processing system of claim 1, wherein the allowed voting marks are at least one of horizontal lines, diagonal lines, vertical lines, arrows, crosses, ticks, and filled target areas. 5. The ballot image processing system of claim 1, wherein the audit data includes at least one of mark categorization, voting mark type and threshold measurements determined by the image processing portion. 6. The ballot image processing system of claim 1, wherein the image processing portion utilizes DIP algorithms to enhance the ballot images to correct for at least one of speckling, dirt, smears and bleed through. 7. The ballot image processing system of claim 1, wherein the image processing portion uses digital image processing techniques to analyze predetermined vote target areas for detection of write-in intent, if write-in intent is detected the image processing portion isolates the ballot image of the predetermined vote target area and associates a sub-image of the scanned write-in with the ballot image record. 8. The ballot image processing system of claim 7, further comprising a display portion and an input portion, wherein if write-in intent is detected, the display portion provides a user with an opportunity to enter the intended write-in via the input portion. 9. The ballot image processing system of claim 8, wherein the entered write-in input data is added to the ballot image record by the printer. 10. The ballot image processing system of claim 9, wherein the entered write-in input data is printed on the ballot by the printer in at least one of a human readable form and a machine readable form. 11. The ballot image processing system of claim 1, further comprising: a display portion; an input portion; and a write-in resolution system that allows the ballot image records and an associated write-in sub-image of the write-in to be viewed by a user through the display portion and verified by the user by typing in the associated name or selecting a registered candidate through the input portion. 12. The ballot image processing system of claim 11, wherein the ballot image records and associated write-in subimage verified by the user are associated with the write-in sub-image and thus the ballot image record. 13. The ballot image processing system of claim 1, wherein the image processing portion analyzes predetermined target areas to detect at least one of poll worker initials or signatures, polling place IDs and precinct IDs.

10

15

20

25

30

35

40

45

50

55

60

65

US 8,864,026 B2 9

10

14.~he ballot image processing system of claim 1, wherein the optical ballot scanner analyzes predetermined target areas to detect the presence of, and decode ID and 2D bar codes. 15. A method of processing a ballot comprising: optically scanning a ballot to create an image of the ballot; detecting, using digital image processing techniques, whether defects due to printing and/or scanning exist in the image of the ballot; correcting any defects on the image of the ballot; identifying target areas on the image of the ballot; categorizing the target areas; identifying whether a voting mark is present in the target areas; determining whether each particular voting mark is valid or not; associating decision information for each voting mark on the ballot; and tallying voting results based on the decision information to supplement the image of the ballot, wherein the supplementing of the image includes at least one color coded highlight indicating target mark categorization and strength of the decision information. 16. The method according to claim 15, wherein the defects include at least one of, printing defects and variations, misfeeds and scanning errors. 17. The method according to claim 15, wherein the target areas are selected from the group consisting of square boxes, rectangular boxes, circles, ellipses, and two arrow ends to be joined. 18. The method according to claim 15, wherein valid voting marks are at least one of horizontal lines, diagonal lines, vertical lines, arrows, crosses, ticks, and filled target areas. 19. The method according to claim 15, wherein the decisiol_linformation includes at least one of mark categorization, votmg mark type and threshold measurements. 20. The method according to claim 15, wherein the decision information is printed on the ballot in at least machine readable form. 21. The method according to claim 15, wherein the decision information is saved as a file associated with an electronic record of the image of the ballot. 22. A ballot image processing system comprising: an optical ballot scanner that scans a ballot to produce an image of a ballot; an image processing portion that processes the image of the ballot with digital image processing techniques by using marks on the ballot to correct for ballot skew and image size variations, the image processing portion being con-

figured to analyze voting target areas on the ballot and make a target decision regarding each mark by identifying specific allowed voting marks on the ballot to assess voter intentions; a printer to print audit data onto the ballot after the ballot has been cast, the audit data including information regarding each target decision made by the image processing portion; and a write-in resolution system that allows the ballot image records and an associated write-in sub-image of the write-in to be viewed by a user through a display portion and verifiedby the user by typing in the associated name ~r selecting a registered candidate through an input portion. 23. The ballot image processing system of claim 22 wherein the ballot image records and associated write-in subimage verified by the user are associated with the write-in sub-image and thus the ballot image record. 24. The ballot image processing system of claim 22 wherein the image processing portion corrects for at least one of printing defects and variations, misfeeds and scanning errors. 25. The ballot image processing system of claim 22 wherein the image processing portion uses digital image processing techniques to analyze predetermined vote target areas for detection of write-in intent, if write-in intent is detected the image processing portion isolates the ballot image of the predetermined vote target area and associates a sub-image of the scanned write-in with the ballot image record. 26. A method of processing a ballot comprising: scanning the ballot to produce an image of the ballot; processing the image of the ballot to correct at least one defect on the image of the ballot, wherein the processing comprises analyzing a target area on the ballot to make a determination regarding a mark on the ballot and assessing voter intent; printing audit data onto the ballot, the audit data including information regarding the determination; and tallying voting results based on the determination of the voter intent and the mark on the ballot to supplement the image of the ballot, the supplementing of the image of the ballot includes at least one color coded highlights indicating target mark categorization. 27. The method according to claim 26, wherein the at least one defect include at least one of printing defects and variations, misfeeds and scanning errors.

10

15

20

25

30

35

40

45

* * * * *

111111

(12)

United States Patent

1111111111111111111111111111111111111111111111111111111111111 US008876002B2

Patent No.: (45) Date of Patent: (10)

Arnao et al. (54) SYSTEMS FOR CONFIGURING VOTING MACHINES, DOCKING DEVICE FOR VOTING MACHINES, WAREHOUSE SUPPORT ANDASSET TRACKING OF VOTING MACHINES

(56)

Subjectto any disclaimer,the term of this patent is extended or adjusted under 35 U.S.C. 154(b)by 0 days.

(21) Appl. No.: 13/092,604 (22) Filed:

235/50R 705/12 235/54R 108/60 52/36.1 312/265.3 710/301 235/51 235/51 235/51 235/51 235/51 235/386 235/386 235/386 726/13 235/386

Primary Examiner - Paultep Savusdiphol (74) Attorney, Agent, or Firm - Holland & Hart LLP

Prior Publication Data US 2012/0048930Al

4/1985 Dolson 2/1987 Boram 3/1987 Carson. 1111991 Mark 9/1997 Sarneret al. 4/1999 Singer 112001 Moon 6/2001 McClureet al. 6/2003 McClureet al. 1112003 McClureet al. 12/2003 McClureet al. 2/2004 McClureet al. 10/2004 Kotobet al 12/2004 McClureet al. 10/2005 Petersenet al. 12/2005 Chunget al. 9/2006 Homewoodet al

(Continued)

Apr. 22, 2011

(65)

References Cited

4,510,378 A * 4,641,240 A * 4,649,264 A * 5,065,832 A * 5,666,765 A * 5,897,180 A * 6,173,352 Bl * 6,250,548 Bl * 6,581,824 Bl * 6,641,033 B2 * 6,662,998 B2 * 6,688,517 Bl * 6,799,723 B2 * 6,827,262 B2 * 6,951,303 B2 * 6,973,581 B2 * 7,111,782 B2 *

(73) Assignee: Dominion Voting Systems, Inc., Denver, CO (US) Notice:

Nov. 4, 2014

U.S. PATENTDOCUMENTS

(75) Inventors: Federico Arnao, Oakland,CA (US); Eric Coomer, Oakland,CA (US);Larry Korb, Oakland,CA (US); Josh Maletz, Oakland,CA (US)

( *)

US 8,876,002 B2

(57)

Mar. 1,2012

Related U.S.Application Data (63) Continuation of application PCT/US2009/062069,filed on Oct. 26, 2009.

No.

(60) ProvisionalapplicationNo. 611193,062,filed on Oct. 24,2008. (51) Int. Cl. G07C 13/00 (2006.01) (52) U.S. Cl. CPC G07C 13/00 (2013.01) USPC 235/386;235/51;235/54F (58) Field of Classification Search USPC 235/51,52,54 E, 54 F, 56, 57, 386, 454, 235/462.01,487,492 See applicationfile for complete searchhistory. 11 VOTING MACHINE

ABSTRACT

A system and device facilitate the storage and tracking of warehousedvotingmachines.A systemincludesa host computer,a pluralityof votingmachinesthat are connectedvia a network to the host computer,each voting machine having one or both of a wireless communicationdevice and a data port that is coupled to the host computer.The system also includesan electionand votingmachinepreparationportion included in the host computerto manage and/or controlthe connectedvotingmachines.The electionandvotingmachine preparationportion is configuredto managethe statusof the connectedvotingmachines,is configuredto instructthe voting machinesto run selftests, is configuredto receiveresults ofthe selftestsbackfromthe connectedvotingmachines,and is configured to prepare/program the connected voting machineswith an electionballot. 17 Claims, 8 Drawing Sheets

US 8,876,002 B2 Page 2

(56)

References

Cited

U.S. PATENT DOCUMENTS 7,392,993 BI D578,564 S 200110013547 Al 200110042005 Al

* * * *

7/2008 10/2008 8/2001 111200I

Prohl et al. ............... 280/33.998 Bolton ............................. D18/6 Kotob et al. .................. 235/386 McClure et al. ................ 705/12

2002/0185395 2003/0047604 2003/0047605 2003/0066872 2004/0169077 2006/0138226 2008/0224582

* cited

Al Al Al Al Al Al Al

* * * * * * *

by examiner

12/2002 3/2003 3/2003 4/2003 9/2004 6/2006 9/2008

Lindamood ................... 206/320 McClure et al. .............. 235/386 McClure et al. .............. 235/386 McClure et al. ................ 235/51 Petersen et al. ............... 235/386 McClure et al. .............. 235/386 Boland ...................... 312/330.1

u.s. Patent

US 8,876,002 B2

Sheet 1 of8

Nov. 4, 2014

11 VOTING MACHINE

HARDWARE INTERLOCKS 1---------.

DISPLAY

22

41

TRACKING DEVICE

NETWORK CONNECTORS

34

28 CPU

32 AUDIO DEVICE 33

r--

INPUT DEVICE

24

FIG. 1

SCANNER

29

PRINTER 30

u.s. Patent

Nov. 4, 2014

US 8,876,002 B2

Sheet 2 of8

9

VOTING MACHINE

11

VOTING MACHINE

11

----I 1 1 I 1 I 1 I 1 I 1 1____

-

---------

_____

HOST COMPUTER

VOTING MACHINE

1---1 1 1 1 1 1 1 1 I 1 1

11

---------

VOTING MACHINE

11

1Q -----1 1 1 1 1 1 1 1 1 1 1 1___

VOTING MACHINE

-

VOTING MACHINE

11

11

FIG.2

u.s. Patent 100~ ~

US 8,876,002 B2

Sheet 3 of8

Nov. 4, 2014

Storage and Maintenance

1-----100

1 101----

Fail

Test

Pass

1Or-

1~

I

Unit Preparation for Election

l...._

1 104~

Fail

Pre-LAT

lpass 10~

Distribution to Polling Locations

1 106----

Acceptance back to warehouse after election

1 '---

107~

Unit Repair

Storage of units in Escrow

FIG.3

r-- 102

u.s. Patent

Sheet 4 of8

Nov. 4, 2014

US 8,876,002 B2

17

18 VOTING MACHINE

21

FIG.4

---11

u.s. Patent

Nov. 4, 2014

US 8,876,002 B2

Sheet 5 of8

35

35

11

11 11

19,20

11 11

11

FIG.5

u.s. Patent

Nov. 4, 2014

17

FIG.6

Sheet 6 of8

US 8,876,002 B2

u.s. Patent

Nov. 4, 2014

Sheet 7 of8

US 8,876,002 B2

I

SIQre Unit in Wim:1bo!d~!il Scan unit 10 and warehouse location Transmit data to Asset Tracking Applic ation

200---

Warehouse Storage

EQ!!!T Unil in!QM2tllJ~1Er~-LAT PrQ~ess Scan unit 10 and delivery location (pre- LAT area) Transmit data to Asset Tracking Applicat ion

201~~

Manual Pre-LAT process ~tag~ 1J0it [~S!QY:fQ[ Q!ili~!:Q! Scan unit 10 and delivery location (Ioeation in staging area) Transmit data to Asset Tracking Applic ation

202~

Staging Area

els!;!i: Uoit 120 deljv!il~ ViilbiQle Scan unit 10 and delivery vehicle 10 Transmit data to Asset Tracking Applic alion

203~

On delivery Vehicle

Q~liv!ilr Unit to Pglling Plak!i! Scan unit 10 and delivery vehicle 10 Enter Pollworker acceptance criteria

204~

At Polling Place

Cclla!.
205-~

On delivery Vehicle

Q~liyer !.ilJiI6S!~~tQ Wa~bQU~~ Scan unit 10 and delivery location (pre -lAT area) Transmit data to Asset Tracking Applic atlon

206~

In Escrow at Warehouse

!;ot~c !.ioil intQAydilLBlilcoYOtPrQQ~~~ Scan unit 10 and delivery location (pre- LAT area) Transmit data to Asset Tracking Applicati on

Audit Unit

207~

FIG.7

u.s. Patent

Sheet 8 of8

Nov. 4, 2014

US 8,876,002 B2

LOCK

45

~41

NETWORK CONNECTOR

DOOR 42

28

FIG.8

f'

41

KEYLOCK SWITCH 43

NETWORK INTERFACE ELECTRONICS 44

NETWORK CONNECTOR

28

FIG.9

US 8,876,002 B2 1

2

SYSTEMS FOR CONFIGURING VOTING MACHINES, DOCKING DEVICE FOR VOTING MACHINES, WAREHOUSE SUPPORT AND ASSET TRACKING OF VOTING MACHINES

machines to be provided with power to operate and charge batteries while being stored in the warehouse, allow voting machines to be connected to a network while being stored in the warehouse, and allow the location of individual voting machines to be tracked while being stored in the warehouse. Another improvement relates to a storage and docking station designed for each type of voting machine and allows voting machines to be stacked and stored safely such that the voting machines are protected from damage. The storage and docking station also is capable of providing power to the machines for battery charging and network connectivity, if supported, for connection to a warehouse management application. The docking station also provides security authentication, which will allow the voting machine to activate various interface ports and support various modes of operation. The protective docking device can accept a voting machine such that the protective docking device provides physical protection for the voting machine while being stored. Additionally, the protective docking device is capable of being stacked on other protective docking devices such that no damage occurs to the voting machines while being stored. Additionally, the protective docking device can be stacked in position either with or without a voting machine attached therein. Another feature of this improvement is that the protective docking device can provide power and communication connections (including network connectivity) to the voting machine. The protective docking device can also provide loop-back connections on I/O ports to support external loop back tests. Further, the docking device can have the necessary authentication devices in them for successful communication with the voting machines. Furthermore, it is within the scope of the improvement that the voting machines can contain location tracking mechanisms such as unique barcodes and RFID tags. Additionally, the plurality of protective docking devices can communicate location information of the voting machine to the asset tracking system. Another improvement provides an asset tracking system that allowsjurisdictions to accurately manage and account for their voting machine assets by allowingjurisdictions to monitor the locations of their voting machine assets both in the warehouse and in transit for an election. For example, the voting machine assets can be scanned when they are staged for shipment, scanned when they are loaded onto a truck or other vehicle, scanned when they are delivered to polling places, scanned when they are picked up from polling places and scanned when they are delivered back to the warehouse. The asset tracking system will then be able to track where the voting machine assets are in that lifecycle. In the asset tracking system, each voting machine can have its own unique serial number identifier, which can be, for example, encoded in a bar code on the outside of the machine. Alternatively, the unique serial number identifier can be encoded in an RFID tag in the machine. Further,the RFID tag in the machine can be an RFID tag that is "read only." Some improvements can further include asset tracking peripheral devices that are capable of reading the serial number identifiers of the plurality of voting machines via bar codes on the outside of the voting machines. In the case where the unique serial number identifiers are encoded in an RFID tag in the voting machines, the asset tracking peripheral devices are capable of reading the serial numbers of a plurality of voting machines via the RFID tags in the voting machines. These peripheral devices can consist of portable

This application is a continuation of International Application No. PCTIUS2009/062069, filed Oct. 26, 2009, which claims the benefit of U.S. Provisional Application No. 611193,062,filed Oct. 24, 2008, each of which are incorporated herein by reference in their entirety.

10

BACKGROUND The warehousing processes associated with an election are some of the most problematic and time consuming in the entire election process, especially for largerjurisdictions. The complexity of these processes increases exponentially with the size of the jurisdiction. Electronic voting systems consist of several disparate systems including the Election Management System (EMS), Ballot Tabulators (digital-optical scan voting machines, direct-record-electronic (DRE) voting machines, etc), as well as other ancillary systems including electronic poll-books, accumulation and consolidation devices, and wireless transmission systems for results. Managing these assets can be a significantburden to jurisdictions. In addition, current voting systems rely on a disconnected process for programming the voting machines to transfer the ballot definition data from the EMS to the voting machines. This is historically accomplished by writing the ballot definition data to a removable memory element from the EMS-flash drive usb drives, secure-digital flash drives, PCMCIA flash drives etc. This disconnected process introduces several failure points in the process, and significantlyincreases the overall effort required of jurisdictions to run an election. Further, due to the periodic nature of elections, voting machines are necessarily stored for periods up to years in between elections. Therefore, it is desirable to produce apparatus for and methods of adequately, safely and efficiently storing voting machines in between elections. Furthermore, in large jurisdictions having several voting machines, it is desirable to provide a means for tracking the voting machines as they are used in an election.

15

20

25

30

35

40

45

SUMMARY In view of the above issues, a number of improvements are presented. The system for configuring voting machines described herein has the following benefits. First, the system significantly reduces the effort required to test the functioning of the voting machine by automating the programming and testing of the machines. Second, the system significantlyreduces the effort required to prepare and conduct pre-election Logic and Accuracy Tests (pre-LAT) on the voting machines, by automating as much of the process as possible in the warehouse. Third, this system allows warehouse workers to identify and locate voting machines that have faults. Fourth, this system allows warehouse workers to identify and locate voting machines that failed to prepare correctly. Fifth, this system allows warehouse workers to identify and locate voting machines that failed pre-LAT.And sixth, this system ensures voting machine integrity by ensuring that network functionality is not available after pre-LAT has been completed. Some improvements allow for the safe stacking and storage of voting machines in a warehouse, allow voting

50

55

60

65

US 8,876,002 B2 3

4

hand held devices containing supporting applications or fixed location devices directly connected to the asset tracking application. One improvement also includes asset tracking applications that are capable of managing and tracking assets utilizing the serial number data collected from a plurality of voting machines that have unique serial number identifiers using a plurality of asset tracking peripheral devices. Further, the asset tracking peripheral devices are capable of communicating the serial number data to the asset tracking application. Another improvement includes a tracking and preparation system for networked voting machines including a host computer, a plurality of voting machines connected via a network to the host computer, each voting machine having one or both of a wireless communication device and a data port for coupling to the host computer, and an election and voting machine preparation portion included in the host computer that is configured to manage and/or control the connected voting machines. The election and voting machine preparation portion can be configured to manage the status of the connected voting machines, can be configured to instruct the voting machines to run self tests, can be configuredto receive results of the self tests back from the connected voting machines, and can be configured to prepare/program the connected voting machines with an election ballot. The self tests run by the voting machines can correspond to pre-LAT tests. The election and voting machine preparation portion can be configured to open pre-LAT polls remotely over the network. The election and voting machine preparation portion can also be configured to run simulation scripts on the voting machines over the network. The election and voting machine preparation portion can further be configuredto disable all network ports of the voting machines after the voting machines have been configured for an election. Each voting machine can contain a location tracking mechanism. The location tracking mechanism can be a barcode and/or an RFID tag, for example. Another improvement relates to a protective docking device for a voting machine. The protective docking device includes a voting machine accepting portion configured to accept and store a voting machine, a power connection portion to provide power to the voting machine stored in the voting machine accepting portion, a receiving portion on a top surface of the protective docking device that is configured to receive another protective docking device stacked thereon, a security authentication portion configured to manage interface ports and modes of operation of the voting machines, and a data connection port to provide a data connection to the voting machine stored in the voting machine accepting portion. The protective docking device can include a plurality of docking stations, each docking station being configured to receive a voting machine. The protective docking device can include a groove on a top surface of each of the docking stations. The docking stations can be stacked in a tiered manner. The protective docking device can include a bag on a back surface of each voting machine within each docking station to collect ballots that have been scarmedby the voting machines. Another improvement relates to a voting machine having an input portion, a network communication device, and a location tracking mechanism.

The voting machine can further include hardware interlocks that disable the network communication device to prevent the voting machine from being accessed via the network communication devices during an election. BRIEF DESCRIPTION OF THE DRAWINGS

10

15

20

25

30

The foregoing and further objects, features and advantages of the invention will become apparent from the following descriptions of exemplary embodiments with reference to the accompanying drawings, in which like numerals are used to represent like elements and wherein: FIG. 1 is a diagram illustrating some of the components of a voting machine; FIG. 2 is a diagram illustrating an example of warehouse networking system; FIG. 3 is a diagram illustrating an example of a warehouse process cycle; FIG. 4 is a diagram illustrating an example of a voting machine in a protective docking station; FIG. 5 is a diagram illustrating an example of a protective docking station in a stacked configuration; FIG. 6 is a is a diagram illustrating an example of a protective docking station in a stacked tiered configuration; FIG. 7 is a flowchartillustrating an example of a process of asset tracking of voting machines; FIG. 8 is a diagram illustrating an example of a hardware interlock; and FIG. 9 is a diagram illustrating another example of a hardware interlock. DETAILED DESCRIPTION OF EMBODIMENTS

35

40

45

50

55

60

65

FIG. 1 illustrates some of the components that can be included in each voting machine 11. The voting machine 11 can include a CPU 32 that controls operation of the voting machine 11 including the functions described herein, a tracking device 34, an audio device 33, an input device 24, an optical scanner 29, a printer 30, network connectors 28 and a visual display unit 22. The network communication device (network connector 28) can be, but is not limited to: ethemet; optical; and wireless communication devices. Voting machine 11 is not limited to these specificcomponents as any number of other components known to one of ordinary skill in the art for inclusion on voting machines 11 could be incorporated therein. Additionally, the voting machine 11 can completely disable the network communication device 28 using hardware interlocks 41. The hardware interlocks 41 prevent the voting machine from being accessed via the network communication devices 28 during an election, for example. Further, the voting machine can run self tests such as, but not limited to: destructivememory tests; non-destructivememory tests; tests ofI/O ports; I/O communication tests; detection of connected peripheral devices; tests of attached peripheral devices; detection of attached Removable Memory Elements (RME); tests of RMEs; and tests of power supplies and batteries (described below). There are a variety of methods that can be employed to hardware interlock the secure RME element. A first implementation is to mount the RME port behind a door 42 that can be locked by a lock 45 and controlled by key access (see, for example, FIG. 8). In addition, a sensor (not shown) can be added to detect whether the door 42 is open or not. If the door 42 is open, an electrical interrupt can be activated to disconnect all power and signal lines connected to the RME port.

US 8,876,002 B2 5

6

FIG. 9 illustrates another embodiment of a hardware interlock 41 where a connection between network interface electronics 44 and network connector 28 is disrupted by a keylock switch 43. To facilitate the preparation of voting machines 11 prior to an election, the voting machines 11 can be remotely instructed to run the self tests mentioned above from over the network and can communicate the results of the tests back over the network. That is, in the warehouse, for example, a plurality of voting machines 11 can be coupled to a host computer over a network using the network connectors 28. The host computer 10 can then control and/or monitor the plural voting machines 11. The network communication device 28 in each voting machine 11 enables the voting machine 11 to be configured and tested remotely. Examples of how voting machines 11 can be configured can include, programming the voting machines 11 with an election ballot over the network, performing validation of a loaded election ballot, and communicating results of that validation back over the network. Additionally, the network communication device 28 enables the voting machines to have polls opened in pre-LAT mode remotely over the network. Pre-LAT polls mode is a standard mode of operation for a voting device for conducting Pre election Logic and Accuracy tests. Further, the communication device enables the voting machines 11 to be provided with a vote simulation script over the network. A vote simulation script is a set of commands that can simulate voting patterns on the machine even to the level of providing precanned scanned ballot images or PDF images of ballots with machine generated marks for testing the vote interpretation engine on the tabulator. The communication device 28 also enables the voting machines 11 to be remotely instructed to run pre-LAT activities such as interpreting vote simulation scripts and images, performing image calibration procedures, verifying all system components for readiness and proper function, self printer test etc. Finally, results of all Pre-LAT tests can be communicated back to the EMS through the communications device 28. Further, the network communication device 28 allows the voting machines 11 to have pre-LAT polls opened remotely over the network, have pre-LAT polls closed remotely over the network and can communicate pre-LAT results back over the network. Additionally, the pre-LAT polls can be closed manually and can communicate pre-LAT results back over the network. Thus, the voting machines 11 can be programmed with an election ballot from over the network, have pre-LAT polls opened remotely over the network and then disable all network ports thru the tabulator firmware and software. Further, the voting machines 11 can be programmed with an election ballot from over the network, have pre-LAT polls opened remotely over the network, have pre-LATpolls closed manually and then disable all network ports. Use of a Network to Prepare Voting Machines Typically the warehouse process cycle consists of the following functions (see FIG. 3): Storage and maintenance 100; Test 101; Repair 102; Machine Preparation 103; Pre-LAT 104; Distribution 105; Acceptance back after election 106; and Escrow storage 107.

The functions listed above, and illustrated in FIG. 3, represent a major logistical effort in large jurisdictions. For example, a jurisdiction has to store its voting machines 11 in a benign environment and keep them charged. Prior to an election, the jurisdiction must test every voting machine 11 to ensure it is operational and repair those that are not. Further, the jurisdiction must then prepare each individual voting machine 11 with the ballot styles for the precinct to which the voting machine 11 is assigned. The preparation and machine function must then be validated against that expected in preLAT.The voting machines 11 that fail the "prc-LAf"process must either be re-prepared or repaired, depending on the issue. The voting machines 11 must then be distributed to the correct locations in a secure manner, where they are used in the election.After the day of election, the voting machines 11 then must be collected and returned to the warehouse and accounted for, where they are stored in escrow (securely) for auditing purposes. The voting machines 11 are finallyretuned to their normal storage modes after auditing or after it is determined that auditing is not needed. For a jurisdiction with just a few voting machines 11, this process is not a major issue. However,for a largejurisdiction, such as ajurisdiction with 5,000 or more voting machines 11, this presents a major logistical problem. Anything that can be done to automate this process and thus reduce the logistical burden on the jurisdictions will be extremely useful. Some of the improvements discussed herein achievethis goal. To solve this problem, the voting machines 11 are networked together to a central management system within the warehouse. Additionally' one improvement includes a software application that assists in the management and implementation of the process. In order to address this logistical problem, all the voting machines 11 are networked together in the warehouse, either with physical network connections or via wireless technology, in their storage positions. Additionally, the voting machines 11 are supplied with power and operate in a special storage mode of operation when in storage. A warehouse management application is used that is capable of sending commands to and receiving data from, the individual voting machines 11, groups of voting machines 11, lists of voting machines or the entire group of voting machines 11 that are stored in the warehouse. Additionally, the warehouse management application is capable of sending a command instructing voting machines 11 to respond by identifying their location in the warehouse and their current status. The networked voting machines 11 then respond by providing the appropriate data. The voting machines 11 are able to report their location by where they are networked and the status information that includes amongst other information, the firmwareversion, battery level, current mode of operation, whether the voting machines 11 have results cartridges present and the current configuration of the voting machines 11. This feature allows the warehouse management application to verify the location of the voting machines 11 and receive information regarding the status of each voting machine 11. Further, the warehouse management application is capable of sending a command instructing the voting machines 11 to run a series of diagnostic tests. The purpose of these tests is to ensure that the hardware is operating correctly. The tests include, but are not limited to the following tests listed below.

Internal Memory Tests

Destructive RAM test

10

15

20

25

30

35

40

45

50

55

60

This tests the RAM by writing data to memory address and reading it back to verify that it has written correctly. It is called destructive because any data residing

US 8,876,002 B2

7

8 -continued in the memory is lost.

Non-Destructive RAM

This tests tbe RAM by writing data to

tests

memory addresses that are not currently in

use and reading it back to verify tbat it has written correctly. Destructive storage memory tests

This tests tbe storage memory (such as CF, hard disk etc.) by writing data to memory address and reading it back to verify that it has written correctly. It is called destructive because any data residing in the memory is

Non-Destructive memory tests

storage

lost. This tests tbe storage memory (such as CF, hard disk etc.) by writing data to memory addresses that are not currently in use and

Removable Memory Element (RME) Tests

Destructive RME tests

reading it back to verify tbat it has written correctly This tests tbe Removable Memory Element (RME) (results cartridge) by writing data to memory addresses and reading it back to verify that it has written correctly. It is called destructive because any data residing in the memory is lost.

Non-Destructive RMW tests

This tests tbe Removable Memory Element (RME) (results cartridge) by writing data to memory addresses that are not currently in

use and reading it back to verify tbat it has written correctly.

Serial Port Tests

Internal tests

These tests test tbe serial ports by performing

internal chip set and internal

loop back tests, by transmitting and receiving data in the various modes

Loop-back tests

supported by tbe chipset. This tests tbe serial ports by perforrning an external loop back tests, by transmitting and receiving data over the port. The serial ports must have a connector that connects the Tx

and Rx lines. Authentication

Tests

This test authenticates

any devices currently

attached tbe serial ports oftbe voting machine.

USB Port Tests

Internal tests

This tests tbe USB ports by performing internal chip set and internal loop back test, by transmitting

Loop-back tests

and receiving data in the

various modes supported by tbe chipset. This tests tbe USB ports by performing an external loop back tests, by transmitting and receiving data over tbe port. The USB ports must have a connector that interfaces the Tx

and Rx lines. Authentication

Tests

This test authenticates

any devices currently

attached tbe serial ports oftbe voting machine.

Etbernet Tests

Internal tests

This tests tbe etbernet port by performing internal chip set and internal loop back tests, by transmitting

Loop-back tests

and receiving data in

tbe various modes supported by tbe chipset. This tests tbe etbernet port by performing an external loop back test, by transmitting and receiving data over Ethernet connection with the warehouse application.

Wi-Fi Tests

This tests tbe Wi-Fi connection by

Internal tests

performing

internal chip set and internal

loop back tests, by transmitting and receiving data in the various modes

Loop-back tests

Authentication

Tests

supported by tbe chipset. This tests tbe Wi-Fi connection by perforrning an external loop back test, by transmitting and receiving data over Wi-Fi connection with the warehouse application. This test authenticates the data encryption

and certificates used in tbe Wi-Fi connection

Otber Port Tests

Internal tests

This tests tbe otber ports by performing internal chip set and internal loop back tests, by transmitting

Loop-back tests

and receiving data in

tbe various modes supported by tbe chipset. This tests tbe otber ports by performing an external loop back tests, by transmitting and receiving data over the port. The ports must

US 8,876,002 B2

10

9 -continued

Authentication

Processor Tests Security Tests Display communication tests Firmware authentication tests

Audit/event logging tests

Printer tests Battery Charging tests Audio sub-system tests ADA device interface tests

Tests

have a connector that connects the Tx and Rx lines. This test authenticates any devices currently attached the other ports on the voting machine. This tests the operation of the processor. These are a suite of tests that test the security features of the voting machine. This tests the connections to the displays and the touch screen membrane. This authenticates the version of the firmware by calculating a HASH value of the firmware image using a seed passed to it by the warehouse application. These tests test the audit and event logging facilities by simulating events and then checking that the events have been written to the logs. This tests the operation of the printer and or VVPAT connected to the voting machine. This tests the battery charging circuits and the state of the battery. This tests the correct operation of the audio sub-system. These test the interfaces provided for the use of ADA devices, for example the audio control unit,

After the tests have been run, the voting machines 11 report back the results of the tests identified above to the warehouse management application. The warehouse management application is then able to identify which voting machines 11 have problems via these tests. This allows the voting machines 11 to be tested remotely without human intervention, thereby reducing the time required to prepare the voting machines for an election. Further,the warehouse management application is capable of setting system parameters in the voting machines 11, such as setting the date and time as well as being capable ofloading election definitions into the machines across the network. Once the election definitions are received, the voting machines 11 authenticate and verify the election definition and copy it to all necessary memory devices including any internal storage devices, and redundant removable memory elements, verifying that it has loaded correctly. The voting machines 11 then report their status back to the warehouse management application. The warehouse management application tracks and manages which voting machines 11 have been prepared successfully and which have had election load issues and thus may require further attention. FIG. 2 illustrates an example of the warehouse management system in a warehouse 9 including host computer 10, and voting machines 11 that are, in this embodiment, connected to the host computer 10 via a wireless network. The host computer 10 includes RAM, ROM one or more CPUs and various interfaces. The warehouse management application is stored on and runs on the host computer 10. This allows the voting machines 11 to be prepared for an election with the election definition automatically without human intervention. The warehouse management application also is capable of loading simulation scripts to the voting machines 11. The voting machines 11 authenticate and verify the simulations scripts and report the status of the load back to the warehouse management application. The warehouse management application is capable of instructing voting machines 11 to open polls in pre-LAT and to run the vote simulation scripts. The voting machines 11 then report the status to the warehouse management application.

30

35

40

45

50

55

60

65

The warehouse management application can also instruct the voting machines 11 to open polls in pre-LATmode, and to accept manual feeding of a deck of test ballots. Once the ballots have been fed into the voting machine 11, a sequence can be initiated on the voting machine 11 to reconnect with the warehouse management application to transmit the results of the test deck for verification and validation. These pre-LAT based processes using either a vote simulation script, or a set of test ballots allows the vast majority of pre-LATto be run automatically without human intervention. Some pre-LAT tests may have to be done manually, such as verifying that: the user interface works correctly; that the scanner mechanism is operating correctly; that test ballots are read correctly; that the audio voting works correctly; and that the printer prints correctly. However, the tests capable of being conducted remotely represent a large reduction in the effort required to prepare the voting machines 11. The warehouse management application is capable of instructing voting machines 11 to close pre-LAT polls and to tally the pre-LAT data. The voting machines 11 then report the pre-LAT tally data back to the warehouse management application. The warehouse management application then compares the pre-LAT data with what was expected to automatically verify that pre-LATwas successfully passed. These measures allow pre-LATto be conducted accurately and with minimum effort. The warehouse management application also is capable of instructing the voting machines 11 to open polls in official election mode and the voting machines 11 then report this back to the warehouse management application. Thus, the entire voting machine preparation and test process can be automated and the required effort to test and prepare the voting machines can be considerably reduced. The warehouse management application also is capable of instructing the voting machines 11 to send their audit and event logs and is being capable of searching for certain types of events. The voting machines 11 can also send back their election tally data (ballot image records) if polls have been closed.

US 8,876,002 B2 11

12

One improvement relates to election and machine preparation applications running on a host computer system connected by a network to a plurality of voting machines 11 that each includes a network communication device via that network (see FIG. 2). One function of the election and machine preparation applications is to manage the status of the connected voting machines 11. Additional functions of the election and machine preparation applications are to instruct the voting machines 11 to rnn self tests (listed below), to receive the results of those self tests back from the voting machines 11, and to display and archive the results. Any errors or issues identified can be communicated back to the user of the system thru these reports and communicated to the warehouse logistics management in order for the machines to be serviced. Another function of the election and machine preparation applications is to prepare/program those voting machines 11 with an election ballot over the network. In addition, the election and machine preparation applications are capable of receiving verification that the ballot has loaded correctly. Further, another function of the election and machine preparation applications is to instruct the voting machines that have an election ballot loaded but have not had any polls opened to open pre-LAT polls. The election and machine preparation applications can then receive verification back that pre-LAT polls were opened successfully. The election and machine preparation applications are also able to provide a vote simulation script to the voting machines 11 that have pre-LAT polls open. The election and machine preparation applications can then receive verification that the simulation script was successfully loaded. The election and machine preparation applications can also suspend operation after pre-LAT polls have been opened so that a set of test ballots can be manually fed into the system. Once the ballot has been fed, the pre-LATpolls can be manually closed and the results of the test ballots commnnicated back to the preparation system for verification. The election and machine preparation applications also can provide those machines 11 that have pre-LATpolls open with instructions to close pre-LAT polls, and the election and machine preparation applications can receive verificationthat the ballots have been validated, and that pre-LAT polls have been closed successfully. The election and machine preparation applications can further provide data on the operational health, pre-LAT data and polls status of the voting machines as well as data on the location in the warehouse of those voting machines Voting Machine Docking and Storage Station A storage and docking station is designed specifically for each type of voting machine 11. This allows the voting machines 11 to be stored safely and protected from damage. Further, the storage and docking station allows the voting machines 11 to be stacked. The storage and docking station also is capable of providing power to the machines for battery charging and network connectivity, if supported, for connection to a warehouse management application. The docking station can also provide connection to support various I/O port loop back tests. The docking station also is capable of providing security authentication, which allows the voting machine to activate various interface ports and support various modes of operation. FIG. 4 illustrates an exemplary embodiment of the storage and docking station 17. As seen in FIG. 4, the storage and docking station 17 includes a cavity 18 into which the voting machine 11 can be placed. FIG. 4 also illustrates some of the plugs and interfaces provided in the storage and docking station 17 for connection with the voting machine 11.As seen in FIG. 4, these connections can include, for example, a

power connection 19 and a data line 20. In some examples,the storage and docking station 17 may comprise a channel 21 provided near a back portion of the docking station 17 to provide a conduit for the cables such as power connection 19 and a data line 20 connected to the voting machine 11. FIG. 5 illustrates how the storage and docking station 17 can be stacked on top of another storage and docking station 17. As seen in FIG. 5, in one embodiment, each docking station 17 can be placed directly on top of the other. Grooves 35 are provided on a top surface of each docking station 17 to facilitate stacking. The grooves 35 are configuredto receive a corresponding projecting portion 36 on the bottom surface of another docking station 17. Additionally, a channel 37 is provided near a back portion of each docking station 17 to provide a conduit for the cables such as a power connections 19 and a data lines 20 connected to each voting machine 11. FIG. 6 illustrates another embodiment of the stacked docking stations 17. In this embodiment, the docking stations 17 are stacked in a tiered manner. Additionally, in this embodiment, the docking stations 17 are configured to allow ballots 1 to be fed into ballot feed trays 38 of the optical ballot scanner 29 in each voting machine 11 while the voting machines 11 are stacked. Therefore, the voting machines 11 do not need to be un-stacked to feed ballots during pre-LAT. Similar to FIG. 4, in the embodiment of FIG. 5, a channel 37 is provided near a back portion of each docking station 17 to provide a conduit for the cables such as a power connection 19 and a data line 20 connected to each voting machine 11. Further, a bag 39 is provided on a back surface of each voting machine 11 to collect ballots that have been scanned. The bags 39 can be disposed on runners 40 so that the bags 39 can slide out to facilitate the removal of the ballots from the voting machines 11. Asset Tracking of Voting Machines Each individual voting machine 11 can be configured with machine-readable identifiers on/in them such as bar codes or RFID devices. These are generally referred to as a tracking device 34 shown in FIG. 1. These machine-readable identifiers can contain information such as, for example, the machine type and serial number. These identifiers can also be used to track the location and 'state' of the voting machines 11 within the election lifecycle. The machine-readable identifiers are capable of being scanned by devices such as barcode scanners and RFID scanners so that the information can be retrieved and used by the tracking and warehouse applications. RFID is preferable as it allows automatic scanning of the deviceswithout the need for manual interaction by a user. When a voting machine 11 is stored in the warehouse, the voting machine 11 is scanned for its identifier information and its location is recorded. If the machine-readable identifier is contained in barcode, then the user will have to scanthat bar code with a bar code reader. If the identifier is contained in an RFID tag, then this can be scanned automatically either by a hand held device or by a scanning device located in the storage area. The location of the voting machine 11 can be inputted in a number of ways. For example, the location of the voting machine 11 could be entered manually by the user, scanned in via a bar code identifying the location, or scanned in via an RFID tag at the location. If the voting machine 11has an RFID tag, and the RFID scanners are located in warehouse storage area, the location of the voting machine 11 can be calculated automatically via comparing the relative strength of the RFID signals or by some other comparative technique. This identification and location information can be automatically passed to the warehouse management and asset tracking systems, preferably via a wireless network. These applica-

10

15

20

25

30

35

40

45

50

55

60

65

US 8,876,002 B2 13

14

tions mark the voting machines as being in a warehouse storage location and record that particular location. Hence the location of a particular voting machine 11 is known and verified. When a voting machine 11 is removed from storage, a similar process occurs. The machine-readable identifier is scanned. Additionally, the reason for the voting machine 11 being moved can be entered into the system, preferably via a button press on a hand held device (which can include multiple selections from which to choose). Therefore, if the voting machine 11 has an RFID tag, the fact that it has been moved from the storage location may be detected automatically.Again, this information can be passed to the asset tracking system, so that its location is still known. If the voting machine 11 is taken to a different location, such as a pre-LAT or audit area (for example to complete the manual aspects of pre-LAT prior to staging for deployment), then the voting machine 11 can be scanned and information such as the voting machines' 11 presence in that different location and that part of the process can be recorded and passed to the asset tracking and warehouse management applications. If thejurisdiction uses a staging area prior to distribution of the voting machines 11,then when the voting machines 11 are placed in that area, the fact that the voting machines 11 are there and the particular location in that area can be recorded using similar means as described above with respect to the storage location at the warehouse. Additionally, vehicles that are used for delivery of the voting machines 11 can also have machine-readable identifiers.Again, these machine-readable identifiers could be stored via barcodes or RFID tags. As voting machines 11 are deployed onto vehicles for delivery,they can be scanned and the vehicle identifier scanned. If the vehicle does not have a barcode or RFID tag, then the identifier could be entered manually. This information can then be relayed back to the asset management application, so that the presence of the voting machines 11 on a particular vehicle can be tracked. In addition, each polling place also can have a machinereadable identifier. These machine-readable identifier could be stored, for example, as barcode or MD tags at the polling place; as codes or barcodes on the delivery sheet; as codes or barcodes in a booklet; or be stored in the hand held device used of scanning for manual selection. If the polling place identifier is contained in a barcode or RFID tag, this can be scanned by the handheld device. If the machine-readable identifier is a code, it can be manually entered in the handheld device. Further, if the machine-readable identifier is stored in the application in the handheld device, then it can be manually selected by the user. When a voting machine 11 is delivered, its ID can be scanned by the handheld device as is the polling place ID. This information is stored in the hand held device. If the process includes an acceptance by someone at the polling place, this can also be recorded in the handheld device (depending on the technology in the hand held device, this could be a signature, a thumb print, an acceptance code orjust a button press). If an attempt is made to transport the voting machine 11 to an incorrect location, the handheld device can identify this and warn the user. When the vehicle returns to the warehouse, the data collected can then be downloaded to the asset tracking application. Hence, the asset tracking application will know what voting machines 11 have been delivered and where the voting machines 11 are located. The asset tracking application also can identify if voting machines 11 have been incorrectly delivered.

When voting machines 11 are picked up from polling locations, the voting machines 11 can be scanned by the hand held device to show they have been collected. Similarly,when the voting machines 11 are placed back into storage or an escrow location, they can be scanned so that the location and this information can be relayed to the asset tracking system. By using this process and information, the asset tracking system can accurately track the location and state of the voting machines 11. Therefore, if a voting machine 11 is mislaid, its path can be investigated to aid finding the voting machine 11. In the warehouse, if a voting machine 11 needs to be retrieved (for example, if it requires repair or is going to be audited) then the asset tracking system can identify its location for easy retrieval. In addition, results cartridges can also have machine-readable identifiers, which can be in the form of barcodes or RFD tags, so that the results cartridges can be tracked in a similar manner as described above. The results cartridges can be scanned when inserted into a machine (which can be a voting machine or another machine) and that information can be relayed back to the warehouse management system so that a specific results cartridge can be associated with a specific voting machine 11. This step is not necessary if there is a networked warehouse management system in use as this can be done automatically via the warehouse management system. Ifresults cartridges are collected separately from the voting machines 11 at the end of the election, then as they are delivered to the tally center, they can be scanned in and tracked. Thus, a record can be kept of which cartridges have been delivered and the time of delivery. FIG. 7 is a flowchart illustrating one example of how the asset tracking process can function. In step 200 the machinereadable identifier of the voting machine 11 is scanned and the location of the voting machine 11 is recorded and transmitted to the asset tracking application. In step 201 the machine-readable identifier of the voting machine 11 is scanned and the delivery location at the pre-LAT area is recorded and transmitted to the asset tracking application. In step 202 the voting machine 11 is ready for delivery and the machine-readable identifier of the voting machine 11 is scanned and the location of the staging area of the voting machine 11 is recorded and transmitted to the asset tracking application. In step 203 the voting machine 11 is placed on a delivery vehicle after scanning the machine-readable identifier of the voting machine 11 and the vehicle ID. This information is then transmitted to the asset tracking application. In step 204 the voting machine 11 is positioned at the polling place after scanning the machine-readable identifier of the voting machine 11 and the delivery vehicle ID. Additionally, when arriving at the polling place the poll worker acceptance criteria is entered. In step 205 the voting machine 11 is placed back on the delivery vehicle after having the machine-readable identifier of the voting machine 11 and the delivery vehicle ID scanned. Once again, this information is then transmitted to the asset tracking application. In step 206 the voting machine 11 is returned back to the warehouse where it is put in escrow.The machine-readable identifierof the voting machine 11 is scannedupon arriving back at the warehouse as well as the delivery location (pre-LATarea). This information is then transmitted to the asset tracking application. Finally,in Step 207 the voting machine 11 enters the audit/recount process. The machine-readable identifier of the voting machine 11 and the delivery location are scanned and the data is transmitted to the asset tracking application. Upon the completion of step 207 the cycle returns back to warehouse storage step 200.

10

15

20

25

30

35

40

45

50

55

60

65

US 8,876,002 B2

15

16

The foregoing description is considered as illustrative only of the principles of the improvements discussed above. The inventions described herein are not limited to specific examples provided herein.

able a.ll network ports of the voting machines after the voting machines have been configured for an election. 7. ~he syste~ accordi~g to claim 2, wherein each voting machine contains a location tracking mechanism. 8. The system according to claim 7, wherein the location tracking mechanism is a barcode. 9. The system according to claim 7, wherein the location tracking mechanism is an RFID tag. 10. The system of claim 1, wherein the election and voting machine preparation portion included in the host computer is fu~her con~gured to. upload vote information from the plurality of voting machines after receiving the cast votes at the polling location. 11. The system of claim 1, wherein the election and voting machine preparation portion is configured to prepare or configure individual voting machines with election-specific information. 12. The system of claim 1, wherein the election and voting machine preparation portion is configured to manage all preelection preparation for the plurality of voting machines prior to deploying the voting machines to a polling location. 13. A voting machine comprising: a processor; an input portion coupled with the processor and configured to receive ballots to be scarmed; a network communication device coupled with the processor and configured to receive programming information from at least one external device; a hardware interlock coupled with the network communication device and configured to disable the network communication device to prevent the voting machine from being accessed via the network during an election period, wherein disabling the network communication device comprises activating an electrical interrupt to prevent network communication with the voting machine while maintaining a physical network connection; and a user interface coupled with the processor and configured to receive cast votes from a voter. 14. The voting machine of claim 13, wherein the voting machine receives all pre-election preparation progrannning information via the network communication device prior to deployment to a polling location. 15. The system according to claim 13, wherein the voting machine contains a location tracking mechanism. 16. The voting machine according to claim 15, wherein the location tracking mechanism is a barcode. 17. The voting machine according to claim 15, wherein the location tracking mechanism is an RFID tag.

What is claimed is: 1.A tracking and preparation system for networked voting machines comprising: a host computer; a plurality of voting machines connected via a network to the host computer, each voting machine comprising: a processor; a user interface coupled with the processor and configured to receive cast votes; a network communication device coupled with the processor, the network communication device comprising a wireless communication device and a data port for coupling the voting machine to the host computer; and a hardware interlock coupled with the network communication device and configured to disable the network communication device to prevent the voting machine from being accessed via the network during an election period, wherein disabling the network communication device comprises activating an electrical interrupt to prevent network communication with the voting machine while maintaining a physical network connection; and an election and voting machine preparation portion included in the host computer that is configured to distribute programming information to each of the connected voting machines prior to receiving the cast votes at the polling location. 2. The system according to claim 1, wherein the election and voting machine preparation portion is configured to manage the status of the connected voting machines, is configured to instruct the voting machines to run self tests, is configured to receive results of the self tests back from the connected voting machines, and is configured to prepare/program the connected voting machines with an election ballot. 3. The system according to claim 2, wherein the self tests run by the voting machines correspond to pre-LAT tests. 4. The system according to claim 2, wherein the election and voting machine preparation portion is configured to open pre-LAT polls remotely over the network. 5. The system according to claim 2, wherein the election and voting machine preparation portion is configured to run simulation scripts on the voting machines over the network. 6. The system according to claim 2, wherein the election and voting machine preparation portion is configured to dis-

10

15

20

25

30

35

40

45

* * * * *

111111

(12)

United States Patent

1111111111111111111111111111111111111111111111111111111111111 US008844813B2

Patent No.: (45) Date of Patent: (10)

Korb et al. (54) ELECTRONIC CORRECTION OF VOTER-MARKED PAPER BALLOT

(56)

US 8,844,813 B2 Sep.30,2014

References Cited U.S. PATENTDOCUMENTS

(75) Inventors: Lawrence Korb, Moraga,CA (US); James Hoover, Outremont(CA)

2005/0247783 2006/0202031 2008/0308634

(73) Assignee: Dominion Voting Systems, Inc., Denver, CO (US) ( *)

Notice:

Al Al Al

*

1112005 Poulos et al. 912006 Chung et al. 1212008 Bolton

235/386

* cited by examiner

Subjectto any disclaimer,the term of this patent is extended or adjusted under 35 U.S.C. 154(b)by 198days.

Primary Examiner - KristyA Haupt (74) Attorney, Agent, or Firm - Holland & Hart LLP

(21) Appl. No.: 13/476,836

(57)

(22) Filed:

Methods, systems, and devices are described for electronically correctingvotes made on voter-markedpaper ballots. An optical scan systemmay image a paper ballot and allow vote modificationby the voter followingthe imaging.Sucha modificationmay be necessary,for example, if a voter has improperly voted (e.g., voted for two candidates for one office),if a definitivedeterminationof the intentionto cast a votecannotbe made (e.g., partiallyfiled-inbubbleassociated with a candidate,erasure,straymarks),if a write-incandidate is includedon the ballot, or if a voterhas changedtheir mind or otherwise wishes to cast a different vote than initially indicated.After correction,the corrected ballot may be reimaged.

(65)

May 21, 2012 Prior Publication Data

US 2013/0306724Al

Nov.21, 2013

(51) Int. Cl. G07C13/00 (2006.01) (52) U.S. Cl. USPC 235/386;235/51;235/454 (58) Field of Classification Search CPC .. G07C 13/00; G07C 13/005; G06K 17/0032; G06K 7/1447; G06K 7/1443; G06K 7/1469; G06K 7/1439 USPC 235/386,51,454 See applicationfile for complete searchhistory.

_____

..iI

38 Claims, 15 Drawing Sheets

)

: Print a record of the confirmed I I write-in on the paper ballot ~ ~

ABSTRACT

J

1435

"-1400

u.s. Patent

Sep.30,2014

Sheet 1 of 15

US 8,844,813 B2

Central Server Computer System 115

110

Optical Scan Ballot System 105

'-

FIG. 1

100

u.s. Patent

Sep.30,2014

US 8,844,813 B2

Sheet 2 of 15

220

~

~ f2Zl

Em

~ Em

City of Anywhere

f2Zl

General Municipal Election January 11891

f2Zl

~ f:2ZI

~ f:2ZI

~

To vote, completely fill in the box(es) BALLOT FOR THE OFFICE

MAYOR

OF

BALLOTFOR THE OFFICE OF

COUNCILLOR Candidate' DO Candidate2 • 0 Candidate 3 D.

VOleforONE(1)1SIchoiCe & ONE {1} 2nd choice, as

Vote for onlyONE(1)candidate

205

•

onext to your choice(s), like this:

shown at right

f2Zl f2Zl f2Zl f2Zl !2'2a f2Zl

210

..... <}-

Em

3) Jacques

~

4) Samuel de

f:2ZI

5) James

IZZl 6)leif I.'ZiZI

7) John

IZZl 8) Henry Em 9) Jake f:2ZI !Z2Zl

10) William L.M. 11)Wilfrid

IZZl 12) JohnA. Em

13) Lester B.

tz;ZJ

14) Louis S.

tz;ZJ

15) Pierre E.

[2Z2l

CHAMPLAIN COOK ERIKSSON FRANKLIN HUDSON JOLIET KING LAURIER MacDONALD PEARSON SAINT·LAURENT TRUDEAU

0

1) Miles J.

0

2) Menelaus P.

0

3) Clause

0

.$

0 0 0 0 0 0 0 0 0 0

"

ru

~

;;::::

MASSEY O'OANIEL VON ESPIE .!:;

.-

.!:d:z

~._

I»

C.

0;'

:J
DO

m


~ ~ffi~ Q ~~ r:::

n

5 _,

nd m",

m

z 'T1 m

r-:

f2Zl

:;;: 0 -0

0

._, Gl

A

0

0

m ;a -e

.." ;tI

c c

:s:

215

s: ....

C)

r-

c :z G1

0000 fCZJ

I.'ZiZI

~

IZ]

fCZJ

""' Z

fCZJ f2Zl

:t: :n rn m

;tI 0
fCZJ IZ]

.::!

~ '" <> '" ~3 '" " ~ '"":;;:

::l

ce

D D

m

~

m

~ ~ ~ ~ ~ ~ ~

"-

FIG. 2

200

u.s. Patent

Sep.30,2014

US 8,844,813 B2

Sheet 3 of 15

~~~~~~m~~~ ORO

City of Anywhere

10\-11 •

General Municipal Election January 11891 To vote, completely fill in the boxles] BALLOT FOR THE OFFICE OF

MAYOR

Vote for on~ ONE(l) candidate

11Robert L. 2) Jean 3) Jacques

4) Samuel de S)James 6) Leif 7)John 8) Henry 9) Jake 10)William l.M. 11)Wilfrid

BORDON CABOT

0 0

o next to your choice(s), like this: BALLOT FOR THE OFFICE OF

2) Menelaus P.

COOK

0 0

ERIKSSON

0

FRANKLIN

0

HUDSON

0

Candidate 2 • 0 Candidate 3

o.

MASSEY

13) Lester B.

.... sa e ~ .!:I :z <- m is: c: ~ '" iil. '"=> :::> -< :::> ~ ~'" "'"'"'s:
."

15) Pierre E.

l

f2Zl f
lZZ3 f2Zl lZZ3

0 KING 0 s: LAURIER 0 0 0 ~ :z: .... s: MacDONALD 0 m z -n 0 n rPEARSON 0 m m c: " 0 :s: .-< C> SAINT·LAURENT 0 TRUDEAU 0 D~D~~D 10 (f)

0 ;>::

0

0

G)

0

::0

Z

0

D0

* "~~ <0-tQ:>

fr

'"

"0

0-t

JOLIET

;:JC

14) Louis S.

f2Zl

D~

O'DANIEL

.$ -rt

~

VON ESPIE

3) Clause

0' ...,0 ..,,-t

n(j "';;0

:x: ::n m m <..> n

s

;0

12) John A,

lZZ3

,0/ '\.~~ 1) Miles J.

CHAMPLAIN

Candidate 1 DO

Vote for ONE(1)lstchoice & ONE (1)2nd choice, as shown at right

~

CARTIER

COUNCILLOR

r;z!

0.

fr s:

» :z: (') m

CD 0

» ::<:J ::<:J

'"

» r 10

m

;0

:s: » z

"'-

FIG. 3

300

u.s. Patent

Sep.30,2014

US 8,844,813 B2

Sheet 4 of 15

110-a Optical Image Module

Network Interface

105-a

415

410

405

Voter Inferface Module 425

Memory

Vote Processing Module 430

420

Vote Record Generation Module 435

Printer

440

"-

FIG. 4

400

u.s. Patent

Sep.30,2014

US 8,844,813 B2

Sheet 5 of 15

540

o ,

Auxiliary Paper

-,

Ballot Path

J

~545

>-- ... - _~-~-530

'" Auxiliary

Tabulator

Ballot Box

Ballot Box

515 510

"-

FIG. 5

500

u.s. Patent

Sep.30,2014

US 8,844,813 B2

Sheet 6 of 15

110-b Memory

Network Interface

605

615

115-a

620

Election Information Collection and Management Module

/'

610

User Interface

625

'-

FIG. 6

600

u.s. Patent

Sep.30,2014

~

US 8,844,813 B2

Sheet 7 of 15

~

~ ~ ~

m

m

~

~ ~ ~

tz:zI

tz:zI

f22ZI E22ZI

lo\_'( I

General Municipal Election January 11891

f22ZI

E22ZI

DRO

City of Anywhere

tz:zI

r2Zl

To vote, completely fill in the box(es)

BAlLOTFOR

BAlLOT FOR THE OFFICE

MAYOR

OF Vote for only ONE (l)candidate 1) Robert L.

BORDON

0

f22ZI f22ZI

710

-

fZZZI.o

r2Zl

COUNCILLOR

THE OFFICE

OF

Candidate1 DO Candidate2 • 0 Candidate3 O.

Vote forONE(1)1stchoice & ONE(1)2nd choice, as shown at right

... ~

H

HUDSON JOLIET

0

KING

0 0 0

ERIKSSON

7) John

FRANKLIN

8) Henry

f22ZI

10)William L.M.

fZ?2J

11)Wilfrid

E22ZI

12)JohnA.

9) Jake

13)Lester B.

LAURIER MacDONALD PEARSON

tz:zI

14)Louis S.

SAINT·LAURENT

E22ZI

15)Pierre E.

TRUDEAU

E22ZI

1

0 0 0 10

.$ 2J ;;:: => =>

.};;

or ... :: -< '<;;:: "" 0

~

:z:

on

"

m ,-

;;0

z .,.,

0

-rr "" 0 0

z

--t (;')

VON ESPIE

DO

!2Zl

-!::' '
~

m

'"

m

;;:: n (") rc

-e

G}

0

s: ;:c

:z:

I2Zl

t;:ZJ

rnI


~ "ffi0'-~

I2Zl

..,

rzza

Q

.,,0

S'

°d m""

....

."

c:

;;:: ):>

z (")

m

(") ):>

I
~ ~ ...=> ~ ~

;;:: 0

0

rn rn

D~

:I: ;;0 m m

m

t;:ZJ

f2Z'l

O'DANIEL

--t

:ll 0

IZZl t;:ZJ

!2Zl

3. ii" '" ::> '" -< !l!..

t;:ZJ

~O

2)MenelausP. 3)Clause

'1,,<:,-1.>

I2Zl

MASSEY

l)MilesJ.

0 0 0 0 0

6) Lei!

E22ZI f22ZI

IZ2Zl

•

o next to your choice(s), like this:

s;2l

f22ZI

~ t;:ZJ

f22ZI

I2Zl

» .-

'"

""

rn

:::0

rzza

s: » :z

;;tJ

~~D~~O

......__

I2Zl

0

rn

V

705

f223

rn

E22ZI

~ ~

~

~ ~ ~ ~ ~ ~ ~ ~ ~

FIG. 7

"-

I- 715

700

u.s. Patent

Sep.30,2014

~

US 8,844,813 B2

Sheet 8 of 15

~

~ ~ ~

m

m

~

m

~

f22Zl

January 11891

fZZl To vole, completely fill in the box(es)

f22Zl

fZZl

BALLOTFOR

fZZl

THE OFFICE OF

I2'Z2'I

Vo!efor only ONE (11candidate

I2'Z2'I

11Robert L

~ fZZl fZZl IZZI

710- a

~ f22Zl

fZZl f22Zl

6) Leif

7)John

8) Henry 9) Jake

fZZI

11)Wilfrid

f22Zl

BORDON

0

H 10)William L.M.

[EJ

MAYOR

o next to your choicets), like this: 8ALLOTFOR THE OFFICE OF

COUNCILLOR Candidate1 00 Candidate2 • 0 Candidate3

Vote forONE(1) 1st choice & ONE(1)2ndchoice,as shownat~ht

12)JohnA. 13)LesterB. 14)Louis S.

fZZl

15)PierreE.

f22Zl

r

""

PEARSON 0

..,

SAINT·LAU RENT 0 TRUDEAU

0

2) MenelausP.

@

f:ZI @

f:ZI

'\,<:-'0

f:ZI

O'DANIELD~

@

~ .£: ~ ~<- ~ rn :;: z " m i;l. :> "'::> ~ ~c: ;g, zs: -<

-

$]

r-'

QI

::> ::>

;s:

f
@

0 0

VON ESPIE

3)Clause

@

~D

MASSEY

1)MilesJ.

0 0 ERIKSSON 0 FRANKLIN 0 HUDSON0 JOLIET 0 KING 0 LAURIER 0 Ma(;DONALD 0

@

o.

....q..

f22Zl

I2'Z2'I

10\-'(1 •

General Municipal Election

f22Zl

f:ZI

ORO

City of Anywhere

f22Zl

~ ~

'"

I2ZJ


~ "ffi ~ 00

Q

~--t

c:

°d m..,

"C

0

I2ZJ f:ZI f:ZI

:r

rn.I

m m

f:ZI

n

I2Zl

-t

;0

:>J 0 en

:;: 0

.... ~ z o

m :z

<"I ;;>;

m r-

0

0

~ ""::> c..

'T1 ;:0

c

0

s: m ::10 -<

c:

:;: n

("J r-'

c:

z Gl

!l>

;;:;:

;po

:z (") m

(")

;po

:;u

:>J

~~O~~D

........_

10

'"'"

» I

rnI

::0

@

» z

@

!S:

705-a

I2Zl

0

m

V

!22a

f22Zl

~

~

~

m ~ ~ m m ~

m~

1--7 15-a

DRO: 2109 Office Of Mayor: 0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,780 Vote for: AMBIGUOUS VOTER CONFIRMED Vote for: Jacques CARTIER

805

./'

Office of Councillor (800,0) (1,790), (0,0) Vote for: Miles J. MASSEY (Ist), Menelaus P. O'OANIEL (2nd) Office of Alderman: 2,812,2,823,788,774 Vote for: OVER·VOTE VOTER CONFIRMED Vole Cor:Jeanne MANCE (1), Nellie McCLUNG (2), Mary PICKFORD (3)

FIG. 8

"-

800

u.s. Patent

Sep.30,2014

US 8,844,813 B2

Sheet 9 of 15

~

~

m

~ ~

m m m

~ ~ ~ ~

rz:zl fZZl

10\-11

General Municipal Election January 11891

£ZZ::l rz:zl

To vote, completely fill in the box(es)

E'Z.Zl E'Z.Zl

BALLOTFOR

MAYOR

THE OFFICE

rz:;zJ

OF

E'Z.Zl

VOte for only ONE (1) candidate

rz:zl

1)RobertL

0

BALLOT FOR THE OFFICE

COUNCILLOR

OF

Candidate! DO Candidate 2 • 0 Candidate3 O.

shown at right

,~

fEiZI

rz:zl

0 0 0

.e: ~ ~ ~

HUDSON

0 0

-=

JOLIET

0

E2ZZl EZ?ZI-- ~6) Leif

ERIKSSON

rz:zl

FRANKLIN

7) John

eEl 8) Henry !Z:2l

I.'i22l

9) Jake 10)WiliiamLM.

rz:zl 11)Wilfrid fZZl rz:zl

12)JohnA. 13)Lester B.

KING LAURIER MacDONALD PEARSON

E'Z.Zl

14) Louis S.

SAINT·LAU RENT

rz:zl

15)Pierre E.

TRUDEAU

rz:zl

I

eEl

~ VOTER CONFIRMED: /' Selection 3

~

920

MASSEY

1)MilesJ.

rz:;zJ

910

•

o next to your choicets), like this: Vote forONE(1) lstchoice & ONE(I) 2ndchoice,as

BORDON

e2ZI

DRO

City of Anywhere

0 0 0

0 0 0

10 ~

O'DANIEL

2) MenelausP.

VON ESPIE

3) Clause -rt OJ

::> ::>

;:: 0>

r-'

<::

~ ~ ~

z

g, Cl;-

.!:' '-

~

m

'"::>::> ~§. ."

'"

'I,,"'~

_,

f2Zl e2ZI e:.zl e2ZI @

e:.zl e:.zl tz2J

D~ DO

I2ZI

f2Zl !2Zl

~"ffi~ Q ~~ <= 0-

IZi3

~D


'C

@

n(j

m""

:r

e;zJ @

rmJ

;<>

m m

!Z:J rz;a

~ 0 (/)

m z m

.,.,

r-

CI

'" s

;:: 0

;0 '1J

n A

'T1

0 ::0 <::I

a.

a;

Z

..... ~ c:. n C') 0 s: rc:: m Z ;0 -< c:.

~D!!Jq

S

I$: Z

C')

m

(")

'" '"

::t>

:;0 :;0

-

~ ... ~

» r-

e2ZI

:.:0

tz::a

» :z

I2ZI

S

VOTER CONFIRMED: Selections I, 4, 9 ~

m

~ ~

m

-s,rrz;a

<,

~ ~ ~ ~

"-

915

1'-9 25

_;

FIG. 9

905

e2ZI

0

m

V

900

u.s. Patent

Sep.30,2014

~

US 8,844,813 B2

Sheet 10 of 15

~

~

m

~

m m

~

m

~ ~

fZZ'I

~ tz;ZJ

1) Robert L 2) Jean

IZ2J

3) Jacques

IZ2J

4) Samuel de

~

5) James

IZ2J

6) Leif

tz:ZI

7) John 8) Henry 9) Jake

~

10)William L.M.

tz;ZJ

11)Wilfrid

tz;ZJ tz;ZJ

-,

MAYOR

OF

s;;zJ

~ ~ f22Z'l

12)John A, 13) Lester B.

o next to your choice(s), 8AlLOTFOR

BAlLOTFOR THE OFFICE

Vote for only ONE {I f candidate

~

1010

To vote, completely fill in the box(es)

IZ2J

IZ2J

lo\,_ y I

General Municipal Election January 1 1891

fZZ'I

~

ORO

City of Anywhere

~

fZZ'I

~ WI

tz;ZJ

0 0 0 0 0

BORDON CABOT CARTIER CHAMPLAIN COOK

0 0

ERIKSSON FRANKLIN HUDSON JOLIET KING LAURIER MacDONALD PEARSON

14)Louis S.

SAINT·LAU RENT

15)Pierre E.

TRUDEAU

) Peter F Jones

0 0 0 0 0 0 0 0

COUNCILLOR

THE OFFICE

OF

r+

.s:; :z

;;:l

I2ZI I2ZI

O'DANIEL

D D~

VON ESPIE

DD

!2ZJ

~._

..

.:;!:

m

I2ZI

"C

IC2a

-I

@

m

ItZI

:x: ::0 m c.>

~

:s: ;0

0 U> m

z

TI

"'0 (")

0 :z -I G)

"

0

;;0

::>C

-rt

m ,.....

0

0

0

s:

I2ZI

'"

/'

1005

:>

Cl.

g: ~

l>

WI

("")

0

rn

IC2a

;J> ::tJ

:;0

:s: l>

Iz:a

z

I2ZI

s:

n '"'

s: z

r-'

;J>

z

n m

m

c:

-<:

Cl

r

;0

O~D~~O

I2ZI I2ZI

I~ ~

mJ

I2ZI


~"'ffi~ Q ~~ <= n2l e m;;o

~ ~

WI

mJ

'" ~'" " ~3
"<

WI

,<:;. ...~'O

2) Menelaus P

;;;:

WI

Candidate3 D.

MASSEY ~

.e .e ~

~

I
1) MilesJ.

3) Clause

!2ZJ

Candidate 1 DO Candidate 2 • 0

Vote For ONE (1)tst choice & ONE (1) 2nd choice, as shownatrilJhI

"T1

•

like this:

WI

~ ~ ~ ~ ~ ~ ~

m

~

PRO: 2109 Office or Mayor: 0,0,1 ,0,0,0,0,0,0,0,0,0,0,0,0,78D Vote for: WRITE-IN VOTER CONFIRMED vote for: Peter F. JONES Office of Councillor (800,0) (1,790), (0,0) Vote for: Miles 1. MASSEY (1st), Menelaus P. O'DANIEL

V (2nd)

Office of Alderman: 2,812,2,823,788,774 Vote [or: Jeanne MANCE (I) , Nellie McCLUNG (2) , Mary PICKFORD (3)

"-

FIG. 10

1000

1015

u.s. Patent

Sep.30,2014

US 8,844,813 B2

Sheet 11 of 15

Receive optical image of a voter-marked paper ballot

-,1105

! Analyze the optical image to determine that one or more votes recorded on the votermarked paper ballot satisfies a criteria for voter clarification

r-, 1110

t

Prompt the voter to provide clarification for the vote(s) recorded on the ballot when one or more votes satisfy the voter clarification criteria

t-, 1115

t Receive clarification for the vote(s) from the voter responsive to the prompting

t Generate a cast vote record responsive to the analyzing and receiving clarification

"

1120

t-, 1125

I I

-----.::t-----1 : Print a record of the received I I clarification on the paper ballot ~ ._ -

-

-

-

-

-

-

FIG. 11

-

-

-

J

1130

~llOO

u.s. Patent

Sep.30,2014

US 8,844,813 B2

Sheet 12 of 15

Prompt voter to insert paper ballot into ballot feeder

1205 1200 Generate optical image of the paper ballot

J

Analyze the optical image to determine vote(s) recorded on the ballot

NO

Prompt the voter to provide clarification or correction for the vote(s) recorded on the ballot

1225 Receive clarification/correction for the vote(s) from the voter responsive to the prompting

1230 Display votes that are to be recorded

1235 Prompt voter to confirm votes that are to be recorded

1240 NO

Generate a cast vote record responsive to the analyzing and any received clarification

1

Print"'ii

1

-----+'1 1250

recordofthereceind-I clarification and/or confirmation on the paper ballot

1 ~

L - - - - - - - - - - J

FIG. 12

1255

u.s. Patent

Sep.30,2014

US 8,844,813 B2

Sheet 13 of 15

Receive optical image of a voter-marked paper ballot

-,1305

1 Analyze the optical image to determine that a write-in vote is recorded on the ballot

-,1310

! Prompt the voter to provide confirmation of the write-in vote

-,1315

1 Receive confirmation for the vote from the voter responsive to the prompting

! Generate a cast vote record responsive to the analyzing and receiving confirmation

"

1320

r-, 1325

I

I I

-----.::t-----I : Print a record of the confirmed 1 1 write-in on the paper ballot ~ ,_ -

-

-

-

-

-

-

-

-

-

J

1330

~1300

FIG. 13

u.s. Patent

Sep.30,2014

US 8,844,813 B2

Sheet 14 of 15

Receive optical image of a voter-marked paper ballot

-,1405

1 Analyze the optical image to determine a write-in vote is recorded on the ballot

t-, 1410

t Prompt the voter to provide confirmation of the write-in vote

r-,1415

! Receive confirmation for the vote from the voter responsive to the prompting

r-, 1420

t Confirm eligibility of the writein candidate

I"-1425

! Generate a cast vote record responsive to the analyzing and confirmations

-,1430

I

I I

-----.:::t-----J

: Print a record of the confirmed J I write-in on the paper ballot ~ ,__-

-

-

-

-

-

-

-

-

-

J

1435

~1400

FIG. 14

u.s. Patent

Sep.30,2014

Sheet 15 of 15

Receive optical image of a voter-marked paper ballot

1505

Analyze the optical image to determine vote( s) recorded on the ballot

1510

YES

YES

US 8,844,813 B2

Prompt the voter to provide confirmation of the vote(s)

1520

Receive confirmation for the vote(s) from the voter responsive to the prompting

1525

Generate a cast vote record including determined votes and any voter confirmations

1535

I I

-----.::t-----J . . c. . C h Pnnt mtormation rrom t e cast J I vote record on the paper ballot ~ J J

._ -

-

-

-

-

-

-

-

-

-

J

1540

~1500

FIG. 15

US 8,844,813 B2 1

2

ELECTRONIC CORRECTION OF VOTER-MARKED PAPER BALLOT BACKGROUND The present inventionrelates to optical scanvoting systems in general and, in particular, to optical scan voting systems including methods and systems for electronically modifying and/or confirming votes in voter-marked paper ballots. Electronic tabulation of voter-marked paper ballots has been used in elections for some time. Such systems may provide efficient vote tabulation if ballots are received without any errors, modifications, or write-in candidates included on the ballot. Such systems may operate through a voter completing a ballot and returning the completed ballot to, for example, a ballot reader for tabulation. In some instances the ballot may not be filled out correctly. Some ballots may also include a write-in candidate's name added by the voter. Such instances may consume election officialresources at the polling place or later. Furthermore, an improperly completed ballot may result in a vote not being counted where; in fact, the voter had desired to cast a vote. Improperly completed ballots may result from a number of circumstances. For example, a voter may vote for two different candidates for a particular office. If such an error is discovered at the polling place, the voter may ask a poll worker to obtain a replacement ballot, which may then be completed. Such a process can be time consuming for both the poll worker and the voter, as specificprocedures may be set out for voiding the original ballot and issuing a replacement ballot. In addition, improperly completed ballots may also include voter response marks that may not be properly tabulated by an electronic tabulating system due to, for example, a voter not completely filling in a bubble indicating a vote, and/or attempting to erase or otherwise cross-out a mark. Each of these cases may result in a ballot that may not be properly counted by the system, and may result in a vote not being counted where; in fact, the voter had desired to cast a vote. Additionally, as mentioned, some voters may choose to write-in a candidate's name for a particular office. Such ballots may include write-in votes in which the names may not be spelled correctly, or handwriting may be illegible. Such situations may result in a ballot that may not be properly counted by the system, and may result in a vote not being counted where; in fact, the voter had desired to cast a vote.

10

15

20

25

30

BRIEF DESCRIPTION OF THE DRAWINGS 35

40

45

SUMMARY Methods, systems, and devices are described for electronically correcting votes made on voter-marked paper ballots. According to some embodiments, an optical scan system is provided in which a paper ballot is imaged and allowed to be modified by the voter following the imaging. Such a modification may be necessary, for example, if a voter has improperly voted (e.g., voted for two candidates for one office), if a definitive determination of the intention to cast a vote cannot be made (e.g., partially filled-in bubble associated with a candidate, erasure, stray marks), if a write-in candidate is included on the ballot, if a voter has changed their mind or otherwise wishes to cast a different vote than initially indicated, or if one or more vote selections are not made on the ballot. According to one set of embodiments, a ballot counting system may receive input from the voter to modify an existing ballot. In some embodiments, a voter takes a completed ballot to an optical scan system, and feeds the ballot into a ballotentry slot. The results of the tabulation may be displayed or

otherwise provided to the voter, and a confirmation may be received that the tabulation is correct. In the event of an error reading the ballot, or an improper vote has been detected (e.g., two votes for one office), the voter may be prompted to provide information to clarify one or more votes, such as by confirming or correcting the vote(s). When confirming or correcting vote information, the voter may also provide an input indicating that a particular vote is to be changed. The system may receive the corrected or modifiedvote, and record the vote in a cast vote record while the voter is still present at the system. In some embodiments, modifications may be printed on the ballot and/or electronically recorded in the cast vote record. Additionally, some voters may elect to write-in a candidate's name for a particular office. Another set of embodiments provides a ballot counting system may receive input from the voter to confirm a write-in candidate vote. In some embodiments, a voter takes a completed ballot to an optical scan system, and feeds the ballot into a ballot-entry slot. The results of the tabulation may be displayed or otherwise provided to the voter, and a confirmationmay be received that the tabulation is correct. In the event that a write-in vote is detected, the voter may be prompted to provide vote clarification that may include confirmation of the write-in vote and write-in candidate selected. The confirmation may include, for example, correct spelling of the write-in candidate's name. The system may receive the corrected or modifiedvote, and record the vote in a cast vote record while the voter is still present at the system. In some embodiments, write-in candidate information may be printed on the ballot and/or electronically recorded in the cast vote record.

50

55

60

65

A further understanding of the nature and advantages of the present disclosure may be realized by reference to the following drawings. In the appended figures, similar components or features may have the same reference label. Further, various components of the same type may be distinguished by following the reference label by a dash and a second label that distinguishes among the similar components. If only the first reference label is used in the specification, the description is applicable to anyone of the similar components having the same first reference label irrespective of the second reference label. FIG. 1 is a block diagram of an optical scan based ballot tabulation system including components configured according to various embodiments. FIG. 2 is an image of a paper ballot according to various embodiments. FIG. 3 is an image of a voter-marked paper ballot according to various embodiments. FIG. 4 is an illustration of an optical scan ballot system according to various embodiments. FIG. 5 is an illustration of an optical scan ballot system and ballot box according to various embodiments. FIG. 6 is a block diagram of a central server computer system according to various embodiments. FIG. 7 is an image of an incorrectly marked paper ballot according to various embodiments. FIG. 8 is an image of an incorrectly marked paper ballot with an appended vote mark indicating electronic voter corrections according to various embodiments. FIG. 9 is an image of an incorrectly marked paper ballot with electronic voter corrections printed on the ballot according to various embodiments.

US 8,844,813 B2

3

4

FIG. 10 is an image of a voter-marked paper ballot with a write-in vote and appended vote mark indicating electronic voter write-in confirmation according to various embodiments. FIG. 11 is a flow chart illustrating operational steps of electronic ballot correction according to various embodiments. FIG. 12 is another flow chart illustrating operational steps of electronic ballot correction according to various embodiments. FIG. 13 is a flowchart illustrating operational steps ofvoter write-in confirmation according to various embodiments. FIG. 14 is another flow chart illustrating operational steps of voter write-in confirmation according to various embodiments. FIG. 15 is a flow chart illustrating operational steps of electronic ballot correction and voter write-in confirmation according to various embodiments.

ing embodiments of the invention. Various changes may be made in the function and arrangement of elements. Thus, various embodiments may omit, substitute, or add various procedures or components as appropriate. For instance, it should be appreciated that the methods may be performed in an order different than that described, and that various steps may be added, omitted or combined. Also, aspects and elements described with respect to certain embodiments may be combined in various other embodiments. It should also be appreciated that the following systems, methods, devices, and software may individually or collectively be components of a larger system, wherein other procedures may take precedence over or otherwise modify their application. One category of electronic voting systems, referred to herein as optical scan systems, receive a paper ballot that has been marked by a voter. The paper ballot is scanned or otherwise imaged to generate an optical image of the ballot, with the optical image then analyzed by analysis software to determine votes that are cast by the ballot. Another category of electronic voting systems, referred to as direct recording electronic (DRE) systems, record votes through a ballot display provided with mechanical or electro-optical components that may be activated by the voter (typically buttons or a touchscreen). In both categories of systems, a tabulation of the voting data is performed, either through a precinct counts provided to a central location, and/or through tabulation performed at a central location. In DRE systems, errors on ballots are limited, as such systems generally do not allow over-votes and incomplete or incorrect voter markings are not enconntered due to binary input from the user interface of the DRE system. Furthermore, write-in candidate names are typed in or selected through some electronic input, thus providing readily legible write-in names. In optical scan voting systems, where a voter marks a paper ballot, errors may be encountered in such instances. For example, an optical scan tabulation system may not record a vote for a particular candidate because a box or bubble next to the candidate's name may not be completely filled in. An election official reviewing such a ballot may readily conclude that the voter intended to cast a vote for the particular candidate, and that the optical scan tabulation system simply did not tabulate the vote because the mark for the candidate did not register as a vote for the candidate. In some situations, however, the intent of the voter may not be clear, and thus a vote may not be counted, or an incorrect vote may be connted. Of course, such instances are not desirable, and aspects of the present disclosure provide systems and methods to receive a confirmationof the voter intent from the voter themselves, while the voter is at the system. This confirmation of voter intent may be included in a cast vote record for the ballot, which is used in tallying election results. With reference now to FIG. 1, a block diagram of a ballot scanning and tabulation, system 100 according to some embodiments is described. In the system 100 of FIG. 1, an optical scan ballot system 105 is configured to receive votermarked paper ballots, scan the ballots, and provide an optical image of the voter marked paper ballots. The optical scan ballot system 105 may include any of several types of scanning equipment, and in an embodiment includes a feeding mechanism that receives ballots and feeds the ballots through a scanner and then into a ballot box. In some embodiments, the optical scan ballot system 105 includes a printer that may mark scanned ballots with one or more marks that may be used to identify the paper ballot. In other embodiments, the optical scan ballot system 105 evaluates marks on the ballots and determines votes made by the voter. The voter may be

DETAILED DESCRIPTION Methods, systems, and devices are described for electronically correcting votes made on voter-marked paper ballots. In some embodiments, an optical scan system is provided in which a paper ballot is scanned or otherwise optically imaged and allowed to be modified by the voter following the scan. Such a modification may be necessary if a voter has improperly voted (i.e., voted for two candidates for one office), if a definitive determination of the intention to cast a vote cannot be made due to, for example, a partially filed-in selection area associated with a candidate, an erasure, stray marks, etc., if a voter has changed their mind or otherwise wishes to cast a different vote than initially indicated, if a write-in candidate is detected on the ballot, and/or if one or more vote selections are not made on the ballot. As discussed, some of these cases could result in the original ballot being voided and the voter given a new ballot. This is a resource intensive process, and also time consuming for the voter. Furthermore, destroying a ballot introduces chain-of-custody issues for the invalid ballots. Various embodiments describe systems and methods for a voter to electronically confirm, clarify, or modify a ballot electronically without the need to complete a replacement ballot. Additionally, as mentioned, some voters may elect to write-in a candidate's name for a particular office. In some embodiments, an optical scan system is provided in which a paper ballot is scanned or otherwise optically imaged and write-in votes are confirmed or otherwise modified by the voter following the scan. Write-in votes may be cast in an optical scan ballot through a target area that is to be marked to indicate the voter has selected to write-in a candidate, and a line is provided for the voter to write the name of the candidate. Capturing these write-in votes may consume a significant amount of resources for election officials, as these votes may need to be reviewed by a human. Furthermore, names may not be spelled correctly, or handwriting may be illegible, which may result in votes not being counted. Additionally, issues may arise when a voter writes a name on a line of the ballot, but has not marked the target area, or where a voter has selected one of the printed candidates and has also written in the write-in area. Various embodiments describe systems and methods for a voter to electronically confirm, clarify, or modify a write-in candidate electronically. This description provides examples, and is not intended to limit the scope, applicability or configuration of the invention. Rather, the ensuing description will provide those skilled in the art with an enabling description for implement-

10

15

20

25

30

35

40

45

50

55

60

65

US 8,844,813 B2 5

6

prompted to clarify the votes determined by the optical scan ballot system 105. In the eventthat one or more vote(s) satisfy a criteria for voter clarification, the voter may be prompted to provide clarification while the voter is still present at the optical scan ballot system 105. The criteria for voter clarification, as will be described in more detail below,may include marginal or ambiguous marks, overvotes, under votes, and/or write-in votes, for example. In some cases, a voter may provide a blank ballot, in which case the voter may be prompted to provide all votes for the ballot. In response to being prompted for clarification, the voter may provide one or more clarifications, such as corrections, confirmations, or modifications of votes. Thus, as used herein, the term "clarification" may include any vote clarifying action, such as corrections, confirmations, or modifications. These clarified votes may be stored in a cast vote record for tabulation to determine election outcome. In some embodiments, confirmed and/or clarified votes may be printed on the ballots by the optical scan ballot system 105 using an internal printer. An optical image of the ballot is obtained by the optical scan ballot system 105, and in some embodiments the vote outcomes determined by the optical scan ballot system 105 are included as an audit mark in the optical image. Such an optical image may then be viewed by other systems to display the optical image of the voter-marked paper ballot along with the audit mark provided by the optical scan ballot system 105. Examples of optical scan ballot systems 105 will be described in more detail below. The optical scan ballot system 105 of FIG. 1 may be connected through a network 110 to a central server computer system 115. Central server computer system 115 may receive cast vote record data from optical scan ballot system 105, and tally votes to determine election results. In some embodiments, a number of optical scan ballot systems 105 are included at a number of different precinct locations. In such cases, each optical scan ballot system may provide a tally of vote outcomes for the particular precinct. In other embodiments' each optical scan ballot system 105 may simply provide cast vote record information to the central server computer system 115. In some embodiments, the central server computer system 115,may receive data containing the optical images of paper ballots and audit marks, if provided, from the optical scan ballot system 105. In some embodiments, the central server computer system 115 may include, as part of the central server computer system 115 or separately, an adjudication system that may allow an election official to review optical images of the paper ballots, and, based on the review, make a determination as to whether the marks were properly counted as votes or whether changes should be made to properly record the votes on the paper ballot. In some embodiments, an audit mark is viewable along with the image of the paper ballot allowing the review of votes recognized by the optical scan ballot system and comparison with the marks on the paper ballot. With reference now FIG. 2, an example of a paper ballot 200 is described. The ballot 200 includes a number of voter selection areas 205, 210, 215 that represent voter selections for different offices or ballot questions. In the example ballot 200, there is also an election official area 220, which commonly appears in ballots, that is used by the election officialto validate the particular ballot. The validation may take the form of a signature or initials in the election official area 220 by the election official,however,other types of authentication may also be used. The election official area 220 may serve to mark the paper ballot 200 as having been officially issued. It will be readily understood that paper ballots such as ballot 200 may include pre-printed ballots and/or ballots printed

locally on-demand by a local printer. The voter may take the validated blank paper ballot 200 to a private area (such as a desk with privacy barriers), and make a mark or marks (this can be done in numerous ways such as filling in a box, making an 'X', etc) beside the chosen candidate(s) with a marking instrument. For the sake of clarity, this disclosure generally refers to selection of a candidate or candidates, however, some votes relate to other questions such as referendum questions, ballot initiatives, and the like. The concepts described herein extend to votes obtained all matters whether candidates, weighted candidate votes, referendum questions and the like. FIG. 3 illustrates an example of a ballot 300 that has been properly completed by a voter.After the voter has marked the ballot 300, the voter provides the ballot to be tabulated. In some situations, the voter may drop the completed ballot (e.g., ballot 300) into a ballot box that election officials then take for processing and tabulation. In some embodiments, the election officials may direct the voter to place the ballot into an optical scan ballot system, such as optical scan ballot system 105 illustrated of FIG. 1, that scans the ballot and performs processing to determine the votes on the ballot. The voter, while still present a the optical scan ballot system, may confirm that the system properly read the votes on the ballot and/or clarify votes on the ballot, as will be described in more detail below. FIG. 4 is an illustration of ballot receiving system 400 according to an embodiment. In this embodiment, the ballot receiving system 400 includes an optical scan ballot system 105-a that a includes a network interface 405 coupled with network 110-a.Network 110-amay be a public network, such as the Internet or Public Switched Telephone Network (PSTN), or may be a private network. The network interface 405 may provide encrypted communications in order to enhance security of the election results. The network interface 405 may include, for example, commonly used network interface hardware to allow connection with and communication over network 110-a. Such network interface hardware may include wired or wireless network interface cards and components, as are well understood in the art. The optical scan ballot system 105-a also includes an optical image module 410, a memory 415 that includes software 420, a voter interface module 425, a vote processing module 430, and a vote record generation module 435. Each of the modules may be implemented using software or hardware, and each may be a means for performing one or more functions as described herein. Components 405-435 may be interconnected through well known mechanisms if needed, such as through one or more data busses, for example. In various embodiments, such as illustrated in FIG. 4, the optical scan ballot system 105-a may also include a printer 440, that may be used to print information on the voter-marked paper ballot. The optical image module 410 may include, or be connected to, an optical imager such as a scarmerthat obtains an optical image of a voter-marked paper ballot. In some embodiments a voter feeds a ballot into a paper feed that is connected to the optical image module 410. Voter interface module 425 may include a display and input device, with instructions to the voter provided through the display. Vote processing module 430 may analyze the optical image and determine votes that are recorded on the ballot. Such analysis may be a pixel-based analysis of voter selection areas on the ballot that are marked by the voter. As mentioned above, in some cases various voter selection areas may not be properly marked or properly read by the vote processing module 430. In the event that the vote processing module 430 determines that an error may be present on the

10

15

20

25

30

35

40

45

50

55

60

65

US 8,844,813 B2 7

8

ballot, or that one or more votes may not have been properly read, the voter may be prompted through voter interface 425 to provide clarification or correction of the vote(s). The determination that an error may be present may be made by the vote processing module 430 when one or more votes recorded on the ballot satisfy a criteria for voter clarification. Such criteria for voter clarification may be established by election officialsprior to an election. In some embodiments, the voter may provide such clarification and/or correction, and the changes may be electronically recorded without the voter having to re-mark the ballot or receive a replacement ballot. Vote record generation module 435 may generate a cast vote record that includes votes determined by the vote processing module and/or modifications/clarifications provided by the voter. As mentioned above, in some cases a voter may select a write-in candidate. In such cases, the presence of a write-in vote may satisfy the criteria for voter clarification. The vote processing module 430 may determine that a write-in vote may be present, and prompt the voter, through voter interface module 425, to confirm the write-in vote and in some cases select the write-in candidate from a list of qualified write-in candidates or otherwise confirm the write-in vote by providing a correct spelling of the candidate's name. In some embodiments, the voter is prompted to confirm that their votes are accurately recorded, and given the opportunity to modify one or more votes. Following the confirmation or modification, the vote record generation module 435 may generate a cast vote record for the ballot. In some embodiments' the vote record generation module 435 may include a printer that prints recorded votes and/or any vote corrections or confirmations on the paper ballot. In such embodiments, a second image of the ballot and printed confirmations and/or corrections may be re-imaged by the optical image module 410. In other embodiments, the vote record generation module 435 may generate an optical image of the vote record, referred to as an audit mark, that may be appended to the optical image of the ballot. Memory 415 may include random access memory (RAM) and read-only memory (ROM), and store computer-readable, computer-executable software code 420 containing instructions that are configured to, when executed (or when compiled and executed), cause the various modules 410-435 to perform various functions described herein (e.g., obtain ballot optical images, receive vote modification/clarification information, generate cast vote records, append an optical image of the audit mark information to the ballot optical image, etc.). The components of the optical scan ballot system 105-a may, individually or collectively, be implemented with one or more Application Specific Integrated Circuits (ASICs) adapted to perform some or all of the applicable functions in hardware. Each of the noted modules may be a means for performing one or more functions related to operation of the optical scan ballot system lOS-a. With reference now to FIG. 5, an example of another optical scan ballot system 105-b is described. The optical scan ballot system 105-b of this embodiment includes a hybrid paper/electronic vote tabulator 505. The hybrid paper/electronic vote tabulator 505, also referred to as a tabulation unit, is coupled with tabulator ballot box 510. An auxiliary ballot box 515 is also illustrated in this embodiment, and may receive ballots that are not provided to the optical scan ballot system lOS-b. As voters come into the polling location, they may be processed by an election official who determines voter eligibility (based on local election rules), and also determines the proper ballot for the voter.The voter may then mark the ballot and take the marked paper ballot to the tabulation unit 505. In some embodiments, the tabulation unit 505

includes a display 520 that instructs the voter to feed the ballot into the tabulation unit 505 through path 525. The ballot may then be fed through the tabulator unit 505 and an image of the ballot created using a digital scanning device 530. According to various embodiments, the scanning device 530 takes a high resolution optical scan of the ballot and moves the ballot into the tabulator ballot box 510 through opening 535. In one particular aspect of the invention, the tabulation unit 505 takes the resulting high resolution scauned image of the entire ballot, and saves a copy of this image to a memory linked to the optical scan ballot system lOS-b. The embodiment of FIG. 5 also includes earphones 540, and a pendant manual trigger 545, for use by voters that require such devices. The tabulation unit 505 of this embodiment includes a processing module that executes software code to analyze the optical image of the ballot to determine the votes recorded on the ballot. The tabulation unit 505, in some embodiments, may also randomly assign a filename to each scanned image to provide that the order in which the ballots were scanned remains private. The tabulation unit 505 may also send a copy of the images to the central server computer system 115 of FIG. 1 for image recognition. In embodiments where tabulation unit 505 performs image processing, a ballot processing application runs an image recognition routine that is applied to the digital image and enables the tabulation unit 505 to selectively recognize specific areas of each image and may analyze such specific areas as described in more detail below to define a series of processing results associated with the particular ballot. For example, the tabulation unit 505 may analyze the various markings on the ballot to ensure it is a valid ballot, perform a pixel count check to verify that the election official area was initialed by the election official, and perform a pixel count of each voter selection area on the ballot. In addition, as further examples of such processing results, depending on the pixel count of each marking box, the mark may be classified as a 'vote', a 'non-vote.' or an 'ambiguous mark.' These classifications may be based, for example, on the total pixel counts of the marking areas (e.g., areas 205, 210, and 215 in the example of FIG. 2) of the ballot, and/or a pattern of pixels in the marking areas. The determination of a vote, non-vote, or ambiguous mark may be made according to pixel levels defined by election officials at a given time prior to the election. In some embodiments, analysis of the optical may include determining that one or more voter selection areas have been selected when markings in the one or more voter selection areas meet a predefined first criteria for determining a specificvoter selection area has been selected. The analysis may also include determining one or more votes recorded on the voter-marked paper ballot satisfy a criteria for voter clarification when markings in one or more voter selection areas meet a predefined second criteria for determining that it is ambiguous whether the one or more voter selection areas has been selected. According to some embodiments, election officials may define, in pixels, the minimum pixel count that is to be classified as a 'vote,' and thus satisfy the predefined first criteria. Election officials may also define, in pixels, the maximum pixel count (if any) that is to be defined as a definite 'nonvote,' and a range of pixels in between those values that will constitute an 'ambiguous mark.' Pixel counts that are within the range of pixels that constitute an ambiguous mark may satisfy the predefined second criteria, and generate a prompt to the voter for clarification of the one or more votes. These pixel values may be loaded on each tabulation unit 505. After calculating these pixel values for each voter selection area, the tabulation unit 505 may return the ballot to the voter with

10

15

20

25

30

35

40

45

50

55

60

65

US 8,844,813 B2 9

10

an appropriate message if any marks on the ballot satisfy the criteria for voter clarification, or feed the ballot to the tabulator ballot box. Criteria for voter clarification may be satisfied, for example, if one or more voting areas contain an ambiguous mark, if too many marking areas in one category were classified as votes resulting in an "over-vote," if no marking areas in one or more category were classified as votes (resulting in a blank ballot for one or more elections) and/or no pixel count was recorded in the election official area. In some embodiments, the tabulation unit 505 may be programmed to allow a voter to verify the ballot in the case of over-voted or blank ballots, thus preserving the voter's right to cast an over-voted or blank ballot. In other embodiments, the optical scan ballot system 105-b may prompt the voter to correct or clarify one or more votes when an ambiguous mark or over-vote is detected, allowing the voter to correct the ballot electronically. The correction(s) may be recoded in a cast vote record generated by the tabulation unit 505. Furthermore, in the eventthat the ballot includes one or more write-in candidates, the tabulation unit 505 may prompt the voter to confirm the write-in candidate and electronically select the write-in candidate name. In some embodiments, once the determination of the total votes for a ballot has been made, the tabulation unit 505 appends an audit mark as a footer to the saved ballot image that contains processing results for that specificballot. In some embodiments, the tabulation unit 505 also includes a printer that may print processing results and any received voter clarifications, and/or corrections on the paper ballot prior to the ballot being moved to the tabulator ballot box 510. Such a ballot may then be re-imaged to provide an optical image of the ballot with printed clarifications and/or corrections. With reference now to FIG. 6, a system 600 is described that includes central server computer system 115-a connected with network 110-b. In this embodiment, the central server computer system 115-a includes a network interface 605, an election information collection and management module 610, memory 615 that includes software 620 stored therein, and a user interface 625. The central server computer system 115-a may communicate with optical scan ballot system(s) such as system 105 of FIGS. 1, and 4-5, through the network 110-b in order to receive and send information to such systems. The network interface 605 may include, for example, commonly used network interface hardware to allow connection with and communication over network 110-b. Such network interface hardware may include wired or wireless network interface cards and components, as are well understood in the art. Each of the modules may be implemented using software or hardware, and each may be a means for performing one or more functions as described herein. Components 605-625 may be interconnected through well known mechanisms if needed, such as through one or more data busses, for example. The election information collection and management module 610 may receive optical image data from optical scan ballot systems and store the data in memory 615. Memory 615 may include a database that is located locally and/or remotely from the central server computer system 115-a. Furthermore, central server computer system 115-a itself may include a single computer, or may include multiple computers which may be located remotely from one another. In any event, the election information collection and management module 610 may receive optical image data and use this data to determine cumulative vote counts for one or more elections, and to perform election management tasks such as, for example, assigning ballots identified as having errors for

adjudication, and assigning ballots to be audited according auditing procedures for the jurisdiction to verify the optical scan ballot systems appear to be properly tabulating votes. In one embodiment, the election information collection and management module 610receives optical image data for each scanned ballot, along with audit mark information for each ballot, and maintains appropriate vote counts based on the data. The election information collection and management module 610 may then store the optical image data, including the ballot optical image and the audit marks, in memory 615 for future retrieval as needed. The user interface 625 may include a monitor to display images to the election management officials, as well as a keyboard, mouse, or other data input device as are well known. An election official accessing the user interface 625 may review optical images including one or more of audit mark information and ballot images, as necessary. The election officialmay also use the user interface 625 to monitor the status of various optical scan systems that may be managed through the system 600. The user interface 625 may also provide statistical information for use by election officials, as well as a number of other types of information that may be useful for efficient election management. Memory 615 may include random access memory (RAM) and read-only memory (ROM), and store computer-readable, computer-executable software code 620 containing instructions that are configured to, when executed (or when compiled and executed), cause the election information collection and management module 610 to perform various functions described herein. The components of the central server computer system 115-a may, individually or collectively, be implemented with one or more Application Specific Integrated Circuits (ASICs) adapted to perform some or all of the applicable functions in hardware. Each of the noted modules may be a means for performing one or more functions related to operation of the central server computer system 115-a. FIG. 7 illustrates an image 700 that includes an optical image of a voter-marked paper ballot 705, that includes improperly marked vote selections. In this example, the ballot 705 includes a target area 710 in the vote for MAYORthat is not completely filled-in, and thus may not be properly recorded as a vote by an optical scan ballot system, such as system 105 of FIGS. 1, and 4-5. Additionally, ballot 705 includes an over-vote 715 for the race of ALDERMAN, where the voter has incorrectly marked four candidates where the instructions state that up to three candidates may be voted for. As mentioned above, an optical scan ballot system may not properly read the vote 710 for MAYOR.In some embodiments' an optical scan ballot system may determine that the mark for the MAYORvote are below a threshold for a vote, and thus indicate that the ballot 705 does not include a vote for MAYOR. In other embodiments, the optical scan ballot system may identify the mark in the MAYOR vote 710 as a marginal or ambiguous mark, as described above, and thus satisfies a criteria for voter clarification. In either event, the optical scan ballot system may prompt the voter based on the determination. Similarly, an optical scan ballot system may properly identify all four votes for the ALDERMAN selection 715, the over-vote satisfying a criteria for voter clarification, and prompt the voter to change one or more of the selections based on the determination. FIG. 8 illustrates an image 800 of ballot 705-a along with an audit mark 805 that may be appended to an optical image 70S-a. As mentioned above, an optical scan ballot system may prompt a voter to clarify or correct the votes 710-a and 715-a that were determined to satisfy the criteria for voter clarification. In response thereto, the voter may confirm that

10

15

20

25

30

35

40

45

50

55

60

65

US 8,844,813 B2 11

12

the vote for MAYOR is to be for selection 3) Jacques CARTIER, and that the Alderman vote for Fanny ROSENFELD 715-awas an error and not a vote. In one embodiment, the audit mark may identify the errors on the ballot and also include an indication that the voter confirmed the different votes, as illustrated in audit mark 805. The cast vote record for the ballot 705-amay also include similar information. Thus, a record of the voter intent is recorded while the voter is still present at the polling place. The audit mark 805, according to some embodiments, is included as an image in the same file that contains the optical image 705-a of the voter-marked paper ballot. In such a manner, if a user displays the optical image of the votermarked paper ballot, the audit mark will also be displayed allowing the user to view the ballot and voter markings as well as information on how the tabulation unit registered the votes for the ballot and any confirmations received from the voter. Such a system allows for auditing of election results in an efficient manner. It will be understood that appending an optical audit mark image to the ballot optical image isjust one manner in which the votes recorded for a ballot may be associated with the ballot, and may be used instead or, or in addition to a cast vote record for the ballot. In some embodiments' the information from the tabulation unit may be stored in a separate database and associated with a particular ballot, or may be stored in the ballot optical image file as extra data or metadata that mayor may not be displayed along with an image of the ballot. The embodiment of FIG. 8 illustrates the audit mark 805 at a footer of the optical ballot image 70S-a, although it will be readily understood that an audit mark image 805 may be located at different locations relative to the ballot image 70S-a.The audit mark 805 may also be referred to as a "fingerprint" or a "vote stamp" that is attached or otherwise associated with each image. In embodiments where the audit mark 805 is appended as an image to the voter-marked paper ballot optical image 705a, the image may be saved in a format that is widely viewable by a variety of different image viewers. For example, the optical image may be stores as a .jpg file, a .tiff'file, or a .pdf file, to name just a few examples. Various different types of viewers may be used to display such files, allowing the review of election results that does not require specialized software. Furthermore, the inclusion of an audit mark with the ballot optical image allows for review of how an optical scanning system read the particular ballot through simply viewing the optical image file. In some embodiments, an audit mark, such as audit mark 805 of FIG. 8, may be printed on the paper ballot by an optical scan ballot system. Such an audit mark may be printed in an open space on the paper ballot that is intended for such a purpose. The ballot with printed audit mark may, in some embodiments, be re-scanned by the optical scan ballot system. In other embodiments, no markings are printed on the paper ballot, with modifications to the cast vote record and/or audit mark associated with the ballot relied on for recording the processing results of the optical scan ballot system and any voter confirmation. In still other embodiments, an optical scan ballot system may print an indication on paper ballots only when voter modification or clarification for a particular vote is received, and such a printed ballot may be re-scanned to generate an optical image of the paper ballot with printed voter modification and/or clarification. FIG. 9 illustrates an image 900 of ballot 905. Ballot 905 includes improper markings 910 for the MAYOR candidate, and 915 for an extra ALDERMAN candidate. In this embodiment, the voter may be prompted to confirm or correct the votes, with confirmation received from the voter. Such confirmation may be

recorded in the cast vote record and/or audit mark for the ballot 905, and in the embodiment of FIG. 9 the optical scan ballot system also prints an indication directly on the paper ballot indicating the voter confirmations. In this example, an indication 920 is printed on the ballot that the voter confirmed selection 3 for the MAYORcandidate, and an indication 925 that the voter confirmed selections 1, 4, and 9 for the ALDERMAN candidates. As mentioned above, in some instances a voter may vote for a write-in candidate for a particular election. FIG. 10 illustrates an image 1000 of a ballot 1005 that includes a write-in vote 1010. In this example, the voter properly marked the target area for a write-in vote, and wrote candidate name Peter F. Jones. The ballot 1005 of this example is otherwise correctly completed, although it will be readily recognized that any incorrectly completed votes may be confirmed as described herein. In this embodiment, an optical scan ballot system may identify that the ballot 1005 includes write-in vote 1010, the presence of a write-in vote satisfying a criteria for voter clarification. The system may prompt the voter to confirmthat they would like to vote for the write-in candidate, and also confirm the name of the write-in candidate. In such a manner, write-in votes may be confirmed and tabulated without the need for an election official to manually read the write-in vote, which may be difficult to read due to poor penmanship, for example. Furthermore, in some jurisdictions, write-in candidates are required to be qualified with election officials prior to the election. In such jurisdictions, the optical scan ballot system may be provided with a list of qualified write-in candidates, and the voter may be notified if their write-in candidate is not included in the list, in which case the voter may decide to vote for a different candidate. The optical scan ballot system may include write-in information in the cast vote record for the ballot and/or include the information in an audit mark for the ballot. In the example of FIG. 10, an audit mark 1015 is appended to the ballot optical image 1005 that indicates that voter confirmed the write-in vote and the candidate that was written in. With reference now to FIG. 11, a method 1100 for electronically correcting voter-marked paper ballots is described. The method 1100 may, for example, be performed by an optical scan ballot system, or a central server computer system, of FIGS. 1, and 4-6, or using any combination of the devices described for these figures. Initially, at block 1105, optical image data comprising an optical image of a votermarked paper ballot is received. As discussed above, optical image data may be received from an optical scan ballot system that scans a voter-marked paper ballot, for example. The optical image is analyzed to determine that one or more votes recorded on the voter-marked paper ballot satisfies a criteria for voter clarification, according to block 1110. Such votes may be identified, for example, as votes that have ambiguous marks on the voter-marked paper ballot, elections or races in which no vote is entered (an 'undervote"), elections or races where too many candidates have been voted for (an 'overvote'), and/or the presence of a write-in candidate. Votes requiring clarification may be identified, in some examples, locally by an optical scan ballot system or locally connected computer, or remotely by a central server computer system, with the optical image data and identified votes for clarification communicated between the central server computer system and the optical scan ballot system over a network. With continuingreference to FIG. 11,the voter is prompted to provide clarification for the vote(s) recorded on the ballot when one or more votes satisfy the voter clarification criteria, as indicated at block 1115. The voter may be prompted, for example, through a user interface located at an optical scan

10

15

20

25

30

35

40

45

50

55

60

65

US 8,844,813 B2 13

14

ballot system. In such embodiments, a voter may feed their ballot into a ballot feed mechanism at the optical scan ballot system, and be prompted to provide clarification for one or more votes before they leave the polling location. At block 1120, clarification for the vote(s) is received from the voter responsive to the prompting. Such clarificationmay include a confirmation of one or more votes, or the correction of one or more votes. For example, if a ballot includes an over-vote,the voter may select the candidate that they would like to vote for, with such a correction received at the system. A cast vote record is generated responsive to the analyzing and receiving clarification, as indicated at block 1125. The cast vote record may include vote information, as well as any clarification information, such as vote confirmations or corrections, that were received from the voter. The cast vote record, according to some embodiments, is used in vote tallying to determine election outcomes. Optionally, as indicated at block 1130, a record of the received clarification may be printed on the paper ballot. The information may be printed on the ballot separately, or as a part of an audit mark that is printed on the ballot, similarly as discussed above.Additionally, in embodiments that include a record of the received clarification printed on the paper ballot, the ballot may be re-imaged to generate second optical image data of the ballot including the printed clarification. In some embodiments, an optical image of the votermarked paper ballot and votes recorded for the ballot may be generated. The optical image data may be provided in a format that is widely known and used, such as, for example, .jpg, .tiff, .bmp, or .pdfformat. Having optical image data in such a format allows the ballot optical image and the adjudication information optical image to be viewed on many different platforms and using any of a number of widely available viewers, rather than requiring a proprietary viewer to view both the ballot and adjudication information. With reference now to FIG. 12, another method 1200 for electronic correction of voter-marked paper ballots is described. The method 1200 may, for example, be performed by a central server computer system or optical scan ballot system of FIGS. 1, and 4-6, or using any combination of the devices described for these figures. Initially, at block 1205, a voter is prompted to insert a paper ballot into a ballot feeder. The ballot feeder may be included as part of an optical scan ballot system and be connected to a ballot box that will ultimately receive the paper ballot. As discussed above, such an optical scan ballot system may prompt the voter to insert the ballot into the feeder, and provide instructions to the voter for completing the voting process. At block 1210, an optical image of the paper ballot is generated. The optical image may be generated from an optical scanner or other imaging device that is connected with the optical scan ballot system for example. The optical image is analyzed, at block 1215, to determine votes that are recorded on the ballot. Such determination may be based on image analysis of the optical image, such as described above, for example. At block 1220 it is determined if clarification of any votes is needed. If clarification is needed, the voter is prompted, at block 1225, to provide clarification or correction for the identified vote(s) recorded on the ballot. At block 1230, clarification and/or correction of identifiedvote(s) are received. Votes that are to be recorded are displayed to the voter, according to block 1235. The voter is then prompted to confirm that votes that are to be recorded, as indicated at block 1240. At block 1245, it is determined if the votes are confirmed. If the voter does not confirm the votes, indicating that they would like to change one or more votes, the operations beginning at block 1225 are performed. If the votes are confirmed at block 1245,

a cast vote record is generated responsive to the analyzing and any received clarifications and/or corrections. Optionally, at block 1255, a record of the received clarifications and/or corrections is printed on the paper ballot. Additionally, in embodiments that include a record of the received clarifications and/or corrections printed on the paper ballot, the ballot may be re-imaged to generate second optical image data of the ballot including the printed information. If, at block 1220, it is determined that clarification is not needed responsive to analyzing of the optical image, operations of block 1235 are performed. In some embodiments, as mentioned above, one or more ballots may include one or more write-in votes. For example, a voter may choose to vote for a candidate that is not listed on the ballot. With reference now to FIG. 13, a method 1300 for electronic confirmation of write-in candidates is described. The method 1300 may, for example, be performed by an adjudication system, a central server computer system, or optical scan ballot system of FIGS. 1 and 4-6, or using any combination of the devices described for these figures. Initially, at block 1305, optical image data comprising an optical image of a voter-marked paper ballot is received. As discussed above, optical image data may be received from an optical scan ballot system that scans a voter-marked paper ballot, for example. The optical image is analyzed to determine vote(s) recorded on the ballot, and to determine whether a write-in vote is recorded on the ballot, according to block 1310. Such votes may be identified, for example, as votes that have marks in a write-in vote target area and/or if analysis of the optical image indicated that writing is present in the write-in vote area. The presence of write-in votes may be identified, in some examples, by a central server computer system, with the optical image data and identified write-in votes for confirmation provided to an optical scan ballot system over a network. With continuingreference to FIG. 13,the voter is prompted to provide confirmation of the write-in vote recorded on the ballot, as indicated at block 1315. The voter may be prompted, for example, through a user interface located at an optical scan ballot system. In such embodiments, a voter may be prompted to confirm that they wish to vote for a write-in candidate, and to enter the name of the write-in candidate. In some embodiments, the writing in the write-in area may be analyzed to determine characters and provide the identified candidate name to the voter for confirmation. In other embodiments, the system may reference a list of registered write-in candidates, and prompt the voter to select a candidate from the list or select a different candidate from the ballot. At block 1320, confirmation of the write-in vote is received from the voter responsive to the prompting. A cast vote record is generated responsive to the analyzing and receiving confirmation, as indicated at block 1325. The cast vote record may include vote information, as well as any clarification information, such as vote confirmations or corrections, that were received from the voter. The cast vote record, according to some embodiments, is used in vote tallying to determine election outcomes. Optionally, as indicated at block 1330, a record of the received confirmation may be printed on the paper ballot. The information may be printed on the ballot separately, or as a part of an audit mark that is printed on the ballot, similarly as discussed above.Additionally, in embodiments that include a record of the received confirmation printed on the paper ballot, the ballot may be re-imaged to generate second optical image data of the ballot including the printed confirmation. In some embodiments, an optical image of the votermarked paper ballot and votes recorded for the ballot, includ-

10

15

20

25

30

35

40

45

50

55

60

65

US 8,844,813 B2 15

16

ing write-in confirmation, may be generated. The optical image data may be provided in a format that is widely known and used, such as, for example, .jpg, .tiff, .bmp,or .pdfformat. Having optical image data in such a format allows the ballot optical image and the adjudication information optical image to be viewed on many different platforms and using any of a number of widely available viewers, rather than requiring a proprietary viewer to view both the ballot and adjudication information. With reference now to FIG. 14, another method 1400 for electronic confirmation of write-in candidates is described. The method 1400 may, for example, be performed by an adjudication system, a central server computer system, or optical scan ballot system of FIGS. 1 and 4-6, or using any combination of the devices described for these figures. Initially, at block 1405, optical image data comprising an optical image of a voter-marked paper ballot is received. As discussed above, optical image data may be received from an optical scan ballot system that scans a voter-marked paper ballot, for example. The optical image is analyzed to determine vote(s) recorded on the ballot, and to determine whether a write-in vote is recorded on the ballot, according to block 1410. Such votes may be identified, for example, as votes that have marks in a write-in vote target area and/or if analysis of the optical image indicated that writing is present in the write-in vote area. The presence of write-in votes may be identified, in some examples, by a central server computer system, with the optical image data and identified write-in votes for confirmation provided to an optical scan ballot system over a network. With continuing reference to FIG. 14, the voter is prompted to provide confirmation of the write-in vote recorded on the ballot, as indicated at block 1415. The voter may be prompted, for example, through a user interface located at an optical scan ballot system. In such embodiments, a voter may be prompted to confirm that they wish to vote for a write-in candidate, and to enter the name of the write-in candidate. In some embodiments, the writing in the write-in area may be analyzed to determine characters and provide the identified candidate name to the voter for confirmation.At block 1420, confirmation of the write-in vote is received from the voter responsive to the prompting. At block 1425, the eligibility of the write-in candidate is verified. As mentioned above, in some jurisdictions write-in candidates are to be qualified prior to an election. In such jurisdictions, various embodiments may provide the ability to reference a list of qualified write-in candidates to confirm eligibility. If the candidate is not qualified, the voter may be prompted to select another candidate, or may simply change their vote to a no-vote. A cast vote record is generated responsive to the analyzing and receiving confirmation, as indicated at block 1430. The cast vote record may include vote information, as well as any clarification information, such as vote confirmations or corrections, that were received from the voter. The cast vote record, according to some embodiments, is used in vote tallying to determine election outcomes. Optionally, as indicated at block 1435, a record of the received confirmation may be printed on the paper ballot. The information may be printed on the ballot separately, or as a part of an audit mark that is printed on the ballot, similarly as discussed above. In some embodiments, an optical image of the voter-marked paper ballot and votes recorded for the ballot, including writein confirmation, may be generated, similarly as discussed above. With reference now to FIG. 15, operations for voter correction and confirmation of votes on a voter-marked paper ballot are described for another set of embodiments. The

method 1500 may, for example, be performed by an adjudication system, a central server computer system, or optical scan ballot system of FIGS. 1 and 4-6, or using any combination of the devices described for these figures. Initially, at block 1505, optical image data comprising an optical image of a voter-marked paper ballot is received. As discussed above, optical image data may be received from an optical scan ballot system that scans a voter-marked paper ballot, for example. The optical image is analyzed to determine vote(s) recorded on the ballot, according to block 1510. At block 1515, it is determined if any errors are present on the ballot. Errors may result from improperly filled in target areas, over-votes, and/or undervotes, similarly as described above. If errors are identified on the ballot, the voter is prompted to provide confirmation and/or clarification/correction of the identified votes. The voter may be prompted, for example, through an interface at an optical scanballot system. At block 1525, confirmationand/or clarification/correction of the identified vote(s) is received from the voter. The voter, in such an instance, may confirm or change the recorded vote, for example, through the voter interface. If, at block 1515, it is determined that no errors are identified on the ballot, it is determined, at block 1530 if any write-in votes are present on the ballot. Such votes may be identified, for example, as votes that have marks in a write-in vote target area and/or if analysis of the optical image indicated that writing is present in the write-in vote area. The presence of write-in votes may be identified, in some examples, by a central server computer system, with the optical image data and identified write-in votes for confirmation provided to an optical scan ballot system over a network. If a write-in vote is present, the operations of blocks 1520-1525 are performed. Ifno errors are identified, and no write-in votes are identified, or following receipt of confirmations and/or corrections to a ballot, a cast vote record is generated, according to block 1535. The cast vote record may include vote information, as well as any correction or confirmation information, such as vote confirmations or corrections, that were received from the voter. The cast vote record, according to some embodiments, is used in vote tallying to determine election outcomes. Optionally, as indicated at block 1540, a record of the received information may be printed on the paper ballot. The information may be printed on the ballot separately, or as a part of an audit mark that is printed on the ballot, similarly as discussed above. In some embodiments, an optical image of the voter-marked paper ballot and votes recorded for the ballot, including write-in confirmation, may be generated, similarly as discussed above. In such a manner, voter-marked paper ballots may be corrected and voter intentions confirmed electronically. In the case of corrections made to a ballot, such corrections may be made electronically through a record in a cast vote record and/or through including the information in an audit mark associated with the ballot, with the information printed on the ballot, according to carious embodiments. It should be noted that the methods, systems and devices discussed above are intended merely to be examples. It must be stressedthat various embodiments may omit, substitute, or add various procedures or components as appropriate. For instance, it should be appreciated that, in alternative embodiments' the methods may be performed in an order different from that described, and that various steps may be added, omitted or combined. Also, features described with respect to certain embodiments may be combined in various other embodiments. Different aspects and elements of the embodiments may be combined in a similar manner. Also, it should be emphasized that technology evolvesand, thus, many of the

10

15

20

25

30

35

40

45

50

55

60

65

US 8,844,813 B2 17

18

elements are exemplary in nature and should not be interpreted to limit the scope of the invention. Specific details are given in the description to provide a thorough understanding of the embodiments. However,it will be understood by one of ordinary skill in the art that the embodiments may be practiced without these specificdetails. For example, well-known circuits, processes, algorithms, structures, and techniques have been shown without unnecessary detail in order to avoid obscuring the embodiments. Also, it is noted that the embodiments may be described as a process which is depicted as a flow diagram or block diagram. Although each may describe the operations as a sequential process, many of the operations can be performed in parallel or concurrently. In addition, the order of the operations may be rearranged. A process may have additional steps not included in the figure. Moreover, as disclosed herein, the term "memory" or "memory unit" may represent one or more devices for storing data, including read-only memory (ROM), random access memory (RAM), magnetic RAM, core memory, magnetic disk storage mediums, optical storage mediums, flash memory devices or other computer-readable mediums for storing information. The term "computer-readable medium" includes, but is not limited to, portable or fixed storage devices, optical storage devices, wireless channels, a sim card, other smart cards, and various other mediums capable of storing, containing or carrying instructions or data. Furthermore, embodiments may be implemented by hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof. When implemented in software, firmware, middleware or microcode, the program code or code segments to perform the necessary tasks may be stored in a computer-readable medium such as a storage medium. Processors may perform the necessary tasks. Having described several embodiments, it will be recognized by those of skill in the art that various modifications, alternative constructions, and equivalents may be used without departing from the spirit of the invention. For example, the above elements may merely be a component of a larger system, wherein other rules may take precedence over or otherwise modify the application of the invention. Also, a number of steps may be undertaken before, during, or after the above elements are considered. Accordingly, the above description should not be taken as limiting the scope of the invention.

2. The method of claim 1, further comprising: printing a record of the received clarification on the votermarked paper ballot. 3. The method of claim 2, further comprising: receiving a second optical image of the voter-marked paper ballot and printed record of the received clarification. 4. The method of claim 2, wherein printing a record comprises: printing information from the cast vote record and a record of the clarificationreceived from the voter in a margin of the voter-marked paper ballot. 5. The method of claim 1, further comprising: generating a vote mark comprising the one or more votes recorded on the voter-marked paper ballot and any clarification for one or more votes received from the voter. 6. The method of claim 5, further comprising: appending an optical image of the vote mark to the optical image of the voter-marked paper ballot. 7. The method of claim 1, further comprising: transmitting the cast vote record to a central server computer system. S. The method of claim 1, further comprising: transmitting the optical image of the voter-marked paper ballot and the cast vote record to a central server computer system. 9. The method of claim 1, wherein the analyzing to determine that one or more votes satisfy the voter clarification criteria comprises: analyzing the optical image to determine whether a writein vote is recorded on the voter-marked paper ballot. 10. The method of claim 9, wherein, the prompting the voter to provide clarification comprises prompting the voter to provide confirmation for the write-in vote; the receiving clarification for one or more votes comprises receiving confirmation for the write-in vote from the voter responsive to the prompting the voter to provide confirmation for the write-in vote; and the generating the cast vote record comprises generating write-in vote confirmation information for inclusion in the cast vote record. 11. The method of claim 9, wherein the prompting the voter to provide clarification comprises: prompting the voter to provide a write-in candidate name. 12. The method of claim 9, wherein the receiving clarification further comprises: determining a write-in candidate included with the write-in vote; and verifying the write-in candidate is qualified with an election authority. 13. The method of claim 1, wherein the voter clarification criteria comprises a write-in vote on the voter-marked paper ballot. 14. The method of claim 1,wherein the voter-marked paper ballot comprises a plurality of voter selection areas, and the analyzing the optical image further comprises: determining that one or more voter selection areas have been selected when markings in the one or more voter selection areas meet a predefined first criteria for determining a specific voter selection area has been selected; and determining one or more votes recorded on the votermarked paper ballot satisfy the criteria for voter clarification when markings in one or more voter selection areas meet a predefined second criteria for determining that it is ambiguous whether the one or more voter selection areas has been selected.

What is claimed is: 1. A method for modifying one or more votes on a votermarked paper ballot, comprising: receiving an optical image of the voter-marked paper ballot; analyzing the optical image to determine that one or more votes recorded on the voter-marked paper ballot satisfy a criteria for voter clarification; prompting the voter on a voter interface to provide clarification for one or more votes recorded on the votermarked paper ballot when one or more votes satisfy the voter clarification criteria; receiving clarification via the voter interface for one or more votes from the voter responsive to the prompting, wherein the clarification comprises at least one of corrections or modifications, of the voter-marked paper ballot; and generating a cast vote record responsive to the analyzing and receiving clarification.

10

15

20

25

30

35

40

45

50

55

60

65

US 8,844,813 B2 19

20

15. The method of claim 14, wherein analyzing the optical image further comprises: performing a pixel count of each voter selection area, the pixel count identifying a number of pixels in the optical image that contain a voter marking. 16. The method of claim 15, wherein the predefined first criteria comprises a minimum pixel count that is to be classified as a vote. 17. The method of claim 15, wherein the predefined second criteria comprises a range of pixel counts between a minimum pixel count that is to be classified as a non-vote and a minimum pixel count that is to be classified as a vote. 18. A system for modifying one or more votes on a votermarked paper ballot, comprising: an optical image module that receives a voter-marked paper ballot and generates an optical image of the votermarked paper ballot; a vote processing module configuredto analyze the optical image to determine that one or more votes recorded on the voter-marked paper ballot satisfy a criteria for voter clarification; a voter interface module configured to: prompt the voter on a voter interface to provide clarification for one or more votes recorded on the votermarked paper ballot when one or more votes satisfy the voter clarification criteria; and receive clarification via the voter interface for one or more votes from the voter responsive to the prompting, wherein the clarification comprises at least one of corrections, or modifications, of the voter-marked paper ballot; and a vote record generation module configured to generate a cast vote record responsive to the analyzing and receiving clarification. 19. The system of claim 18, further comprising: a printer coupled with the vote record generation module and configured to print, on the voter-marked paper ballot, a record of the received clarification for one or more votes from the voter. 20. The system of claim 19, wherein the printer is further configuredprint an identificationof the one or more votes that were determined to require clarification and a record of the clarification received from the voter in a margin of the votermarked paper ballot. 21. The system of claim 20, wherein the optical image module further generates an optical image of the votermarked paper ballot and printed clarification. 22. The system of claim 19, wherein the vote processing module is further configured to generate a vote mark comprising the one or more votes recorded on the voter-marked paper ballot and any clarification for one or more votes received from the voter; and wherein the printer is further configured to print the vote mark on the voter-marked paper ballot. 23. The system of claim 18, wherein one or more votes satisfy the voter clarification criteria when a write-in vote is recorded on the voter-marked paper ballot. 24. The system of claim 23, wherein the voter interface module is further configured to prompt the voter to provide confirmation for the write-in vote and receive confirmation of the write-in vote; and wherein the cast vote record includes write-in vote confirmation information. 25. The method of claim 23, wherein voter interface module is further configured to prompt the voter to provide a write-in candidate name.

26. The system of claim 18, wherein the paper ballot comprises a plurality of voter selection areas, and the vote processing module is further configuredto perform a pixel count of each voter selection area, the pixel count identifying a number of pixels in the optical image that contain a voter marking. 27. The system of claim 18, wherein the vote record generation module is further configured to generate a vote mark comprising the one or more votes recorded on the votermarked paper ballot and any clarification for one or more votes received from the voter. 28. The system of claim 27, wherein the vote record generation module is further configured to append an optical image of the vote mark to the optical image of the votermarked paper ballot. 29. The system of claim 18, further comprising: a network interface coupled with the vote record generation module and configured to transmit the cast vote record to a remote system. 30. A computer program product, comprising: a non-transitory computer-readable medium comprising: code for receiving an optical image of the voter-marked paper ballot; code for analyzing the optical image to determine that one or more votes recorded on the voter-marked paper ballot satisfy a criteria for voter clarification; code for prompting the voter on a voter interface to provide clarification for one or more votes recorded on the voter-marked paper ballot when one or more votes satisfy the voter clarification criteria; code for receiving clarificationvia the voter interface for one or more votes from the voter responsive to the prompting, wherein the clarification comprises at least one of corrections, or modifications, of the votermarked paper ballot; and code for generating a cast vote record responsive to the analyzing and receiving clarification. 31. The computer program product of claim 30, wherein the computer readable medium further comprises: code for printing, on the voter-marked paper ballot, a record of the clarification received from the voter. 32. The computer program product of claim 31, wherein the computer readable medium further comprises: code for receiving a second optical image of the corrected voter-marked paper ballot. 33. The computer program product of claim 30, wherein the computer readable medium further comprises: code for generating a vote mark comprising the one or more votes recorded on the voter-marked paper ballot and any clarificationfor one or more votes received from the voter. 34. The computer program product of claim 33, wherein the computer readable medium further comprises: code for appending an optical image of the vote mark to the optical image of the voter-marked paper ballot. 35. The computer program product of claim 30, wherein the computer readable medium further comprises: code for transmitting the cast vote record to a central server computer system. 36. The computer program product of claim 30, wherein the computer readable medium further comprises: code for transmitting the optical image of the voter-marked paper ballot and the cast vote record to a central server computer system.

10

15

20

25

30

35

40

45

50

55

60

65

US 8,844,813 B2

21 37. The computer program product of claim 30, wherein the voter-marked paper ballot comprises a plurality of voter selection areas, and the code for analyzing the optical image further comprises: code for determining that one or more voter selection areas have been selected when markings in the one or more voter selection areas meet a predefined first criteria for determining a specific voter selection area has been selected; and code for determining that one or more votes recorded on the voter-marked paper ballot satisfy the criteria for voter clarification when markings in one or more voter selection areas meet a predefined second criteria for determining that it is ambiguous whether the one or more voter selection areas has been selected. 38. The computer program product of claim 37, wherein the predefined second criteria comprises a range of pixel counts between a minimum pixel count that is to be classified as a non-vote and a minimum pixel count that is to be classified as a vote.

* * * * *

22

10

15

20

111111

(12)

US008910865B2

United States Patent

(10)

Coomer et al.

(45)

(54) BALLOT LEVEL SECURITY FEATURES FOR OPTICAL SCAN VOTING MACHINE CAPABLE OF BALLOT IMAGE PROCESSING, SECURE BALLOT PRINTING, AND BALLOT LAYOUT AUTHENTICATION AND VERIFICATION (75) Inventors: Eric Coomer, Broomfield, CO (US); Larry Korb, Moraga, CA (US); Brian Glenn Lierman, Exeter, CA (US) (73) Assignee: Dominion Voting Systems, Inc., Denver, CO (US) ( *)

1111111111111111111111111111111111111111111111111111111111111

Notice:

Subject to any disclaimer, the term of this patent is extended or adjusted under 35 U.S.C. 154(b) by 0 days.

Apr. 22, 2011

(65)

Jun. 14,2012

Related U.S. Application Data (63) Continuation of application PCT/US2009/061343, filed on Oct. 20, 2009.

No.

(60) Provisional application No. 611193,062,filed on Oct. 24,2008. (51) Int. Cl. G07C 13/00

(52) U.S. Cl. CPC USPC .

Dec. 16,2014

(58) Field of Classification Search USPC 235/386 See application file for complete search history. (56)

References Cited U.S. PATENTDOCUMENTS

3,542,287 A 7,306,148 Bl 2005/0052519Al 2006/0081706Al 2007/0170253Al 2008/0093455Al 2008/0110985Al

* * * * * * *

1111970 Margaretoset al. 12/2007 Morganstein 3/2005 Mayeret al. 4/2006 Onischuk 7/2007 Chunget al. 4/2008 Barten 5/2008 Cohenet al.

235/386 235/386 347/105 235/386 235/386 235/454 235/386

* cited by examiner

(57)

Prior Publication Data US 2012/0145784 Al

US 8,910,865 B2

Primary Examiner - Thien M Le Assistant Examiner - Toan Ly (74) Attorney, Agent, or Firm - Holland & Hart LLP

(21) Appl. No.: 13/092,599 (22) Filed:

Patent No.: Date of Patent:

(2006.01)

ABSTRACT

A ballot authentication system uses a plurality of security features embedded in and/or printed on the paper stock used to print a ballot on which election-choice-information is printed and a voting unit that includes at least a scanner that is configured to detect the plurality of security features that are embedded in and/or printed on the ballot and authenticate the ballot based on the read information. The voting unit of the ballot authentication system can be configured to verify and confirmthat the various security features embedded in and/or printed on the ballot is correct for a particular precinct of an election. The security features of the ballot authentication system can include static, dynamic and data security features.

G07C 13/00 (2013.01)

....................... 235/386

DEMONSTRATION

11 Claims, 5 Drawing Sheets

BALLOT

Famous Names 100 wI Straight Party INSTRUCTIONS TO VOTERS: ToVOTE,CONNECTTHEARROW like this......... Tovcte for a write-ill candidate,write the person's and CONNECTTHEA.RROW .......

•

u.s. Patent

Dec. 16,2014

Sheet 1 of 5

Y"//"~~~~ //////////////////////////////////////////////////////////////////////////////////////. '//////////////////////////////////////////////////////////////////////////////////////.

8

\

W///"///"//"$//"~~$$//"~~///j///"//"///j///"~//"//"~ ///////////////////////////////////////////////////////////////////////////////////////. ///////////////////////////////////////////////////////////////////////////////////////

8

\

W$//"//"$////"~$/$//"~$//"///"~//"//"//"~ //////////////////////////////////////////////////////////////////////////////////////. //////////////////////////////////////////////////////////////////////////////////////

FIG. 1

US 8,910,865 B2

u.s. Patent

Dec. 16,2014

Sheet 2 of 5

0-'ffffh0'ff/ffffff&0:/$ff$~ff/$/&0:/ff/ffffff//wffffff/ff~h/////////////////////////////////////////////////////////////////////////////////////// v//////////////////////////////////////////////////////////////////////////////////////

\, 8'

\

0-'ffh0'$ffffffff$ff$ffffff/ffff$ffffffff$ffff$/ffffffffffffffffff/ffff/h////////////////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////////////////

8'

\

WW$ff$ff/ffffff$/$$ff/ffffff$$ffffff$/ff/$/ffffffffff/ff/ffffh////////////////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////////////////

FIG.2

US 8,910,865 B2

u.s. Patent

Dec. 16,2014

US 8,910,865 B2

Sheet 3 of 5

11

r

1

38

FIG.3

u.s. Patent

Dec. 16,2014

US 8,910,865 B2

Sheet 4 of5

11 VOTING UNIT DISPLAY

22

TRACKING DEVICE

NETWORK CONNECTORS

34

28 CPU

32 AUDIO DEVICE

....__

r---

SCANNER

29

33

INPUT DEVICE

24

FIG.4

PRINTER

30

u.s. Patent

Dec. 16,2014

3

Sheet 5 of 5

US 8,910,865 B2

DEMONSTRATIONBALLOT Famous Names 100 wI Straight Party

2

5

4 6

1

ZACHARYTAYLOR& MILLARDFILLMORE '/IIglniaParly

BENJAMINHARRISON& ADLAI E. STEVENSON OhIOParty

..

..

ASSOCIATE JUSTICE OF THE SUPREME COURT

....

STATE SENATOR 371h DISTRICT Vote For One

Vote for One

LEARNEDHAND Vrrgir'ta.P"'t~

....

Write· In

DIRECTOR OF ENTERTAINMENT

...... .... ......

...... ...... .. .... .. .... .... .. .. .... .. .."~;;;;~v~~~.~~~r~o~n·~~~Jr.~::~~~~::~ __ l t============== ~ .. .. .. .. .. .... .... .. .. .. .. .. .. Vote for Three

U.S. SENATOR Vote For One

EVERETTDIRKSEN VirginiaP;tft,

CHARLESCURTIS Ohio Party

JOHNHANCOCK CaliforniaPariy

NELSONW. ALDRICH Ntl'IlYor~P .. 1y

WIIIE,ln

•

Wrile-In

..

..

MEMBER OF STATE LEGISLATURE 3rd DISTRICT

.. SUSANB. ANTHONY IArglfllll P;:,rty

MAMIEEISENHOWER

DIRECTOR OF TRANSPORTATION

•

FIG.5

US 8,910,865 B2 1 BALLOT LEVEL SECURITY FEATURES FOR OPTICAL SCAN VOTING MACHINE CAPABLE OF BALLOT IMAGE PROCESSING, SECURE BALLOT PRINTING, AND BALLOT LAYOUT AUTHENTICATION AND VERIFICATION This application claims the benefit of U.S. Provisional Application No. 611193,062filed Oct. 24, 2008. The disclosure of U.S. ProvisionalApplication No. 611193,062is incorporated herein by reference in its entirety.

2

5

10

BACKGROUND The improvements described herein relate to technologies for secure ballot image processing, ballot printing, and ballot layout authentication and verification. Of great importance in maintaining the integrity of the voting process is ensuring that only authentic ballots are used during an election. In addition, due to the disconnected nature of optical scan based voting systems (in which an optical scanner is used to interpret voter intent and tabulate paper ballots that were previously filled-outby voters), it is imperative that the system can identify and verify that the content of the printed ballot matches the electronic definition that the system uses to interpret and process the ballot. In this regard, it is desirable to develop a ballot that includes certain security features to deter unauthorized printing, copying or counterfeiting of the ballot, as well as secure identifying information for the ballot layout.

15

20

25

30

SUMMARY In view of the above issues, a number of improvements are presented. Some improvements relate to layered security features for ballots and to a ballot authentication system for both precinct and central optical ballot scanners. Particularly, improvements relate to the variable combination of latent security features in every ballot. The security features can be readable by embedded sensors in the optical ballot scanners. Such features prevent unauthorized, duplicated, and/or counterfeit ballots from being counted as valid ballots. Further, these features ensure the ability to track a ballot from generation to tabulation, thereby ensuring a secure chain of custody from beginning to end, and the ability to fully audit the life cycle of a given ballot. Finally, invalid ballots can be clearly marked utilizing an integrated ballot imprinter to clearly identify counterfeit, duplicated, or unauthorized ballots. Some improvements provide a secure system for the production, printing, inspection, and authentication of ballots used in an election. Further, such improvements can prevent the unauthorized generation, printing, duplication, or counterfeiting of ballots for use in an election. Some improvementsrelate to a ballot layout authentication system for precinct and central optical ballot scanners. Particularly, such improvements relate to authentication features that help to guarantee that a printed ballot matches the electronic definition of the ballot used by the optical ballot scanners to process and interpret the voter marks on the paper ballot Some improvements provide a validation mechanism for verifying that the electronic definition of the ballot layout matches the physical printed ballot. This validation mechanism will ensure that the disparate definitions are in sync and thus will ensure the integrity of the ballot interpretation, and correct tally and tabulation of the voter-marked ballots.

35

40

45

50

55

60

Ballots, such as paper ballots, on which election choice information is printed (that is, one or more items for which a voter is to cast hislher vote (the items can request a voter to choose/select a candidate for a particular officeand/or request the voter to vote for or against a proposal/referendum, etc.)) contain one or more security features to be described in more detail below. A ballot authentication system can include the above-mentioned ballots and a voting unit that processes the ballots. The ballots can include a plurality of security features that are embedded in paper stock used to print the ballots. A plurality of security features also can be printed on each ballot during the process of printing the ballot. The voting unit can include at least an optical ballot scanner that is capable of detecting and verifying the plurality of security features embedded in the paper stock used to print the ballot and the plurality of security features printed on the ballot during the process of printing the ballot. The voting unit can be configuredto verify and confirm (authenticate) the various security features embedded in and printed on the ballots. The security features can include, static, dynamic and data security features. The security features can include at least one of ultraviolet features, infra-red features, magnetic features, fluorescent features, visual ink features and watermarks. The data security features can include at least one of plain and encrypted data. At least some of the security features may be masked by one another. For example, a printed security feature can be printed on the ballot over a security feature that is embedded in the paper stock used to make the ballot. A further aspect provides a method of validating and authenticating a ballot. The method includes calculating a unique authentication value based on election information provided on the ballot (such as the given set of contests and candidates positioned on the ballot), printing the unique authentication value on the ballot, providing an optical ballot scanner that is configured to receive ballots having the authentication value printed thereon, comparing the authentication value provided on the ballot (as scarmedby the optical scanner) with an authentication value storedby the optical ballot scanner,and marking the ballot as invalidif the scarmed authentication value does not match the authentication value stored by the optical ballot scarmer. Another aspect provides a method for authenticating ballots used in an election having multiple precincts. The method includes (i) providing a plurality of ballots on which electionchoice-information is printed, the ballots having a plurality of security features; (ii) providing, from among the plurality of ballots, a first set of ballots having a first set of the plurality of security features in each ballot; (iii) assigning the first set of ballots to a first precinct; (iv) providing, from among the plurality of ballots, a second set of ballots having a second set of the plurality of security features in each ballot, the second set of security features being different from the first set of security features; (v) assigning the second set of ballots to a second precinct that is different from the first precinct; (vi) confirming, after a vote has been cast, whether a particular ballot has the first set of security features or the second set of security features and whether the particular ballot was cast in the first precinct or the second precinct; and (vii) marking the particular ballot as invalid if the particular ballot does not have the set of security features from the precinct in which the ballot was cast. BRIEF DESCRIPTION OF THE DRAWINGS

65

The foregoing and further objects, features and advantages of the invention will become apparent from the following

US 8,910,865 B2 3

4

descriptions of exemplary embodiments with reference to the accompanying drawings, in which like numerals are used to represent like elements and wherein: FIGS. 1 and 2 are diagrams illustrating examples of ballot security features; FIG. 3 is a diagram illustrating an example of a voting unit; FIG. 4 is a diagram illustrating some of the components of a voting unit; and FIG. 5 is a diagram illustrating an example of a ballot.

The printed security features 8' of FIG. 2 can be printed separately from or at the same time that the election choice information is printed. It is preferred that embedded and printed security features be used. FIG. 3 illustrates an example of a voting unit 11 that includes an optical ballot scanner 29 (see FIG. 4). As seen from FIG. 3, voting unit 11 can include an input slot 23 into which a ballot 1 to be scanned is fed, a ballot feed tray 38, a display 22, an audio device 33 (having speakers), and a usermanipulatable input device 24. FIG. 4 illustrates some of the components that can be included in each voting unit 11. The voting unit 11 can include a CPU 32 that controls operation of the unit 11 including the functions described herein, a tracking device 34, an audio device 33, an input device 24, an optical scanner 29, a printer 30, network connectors 28 and a visual display unit 22. Voting unit 11 is not limited to these specific components as any number of other components known to one of ordinary skill in the art could be incorporated therein. After a voter fills-ina ballot, the voter inserts the completed ballot into the slot 23 of the voting unit 11. The voting unit 11 then optically scans the ballot with its internal scanner 29, which can be a CCD scanner, for example. An image of the scanned ballot then can appear on the display 22. By viewing the image, the voter can confirm that the ballot image is correct. In addition, by using image recognition technology (see, for example, U.S. Pat. No. 6,854,644 to Bolton et al., the disclosure of which is incorporated herein by reference in its entirety), the voting unit 11 determines the selections made by the voter on the ballot (i.e., determines which candidates, etc. were selected by the voter) and displays those determined selections to the voter via display 22. The user can then confirm that the voting unit's determinations are correct. Once confirmed, the voting unit's determinations are stored in memory for future tabulation. The ballot 1 also is stored in the voting unit 11. The voting unit 11 also is capable of detecting and verifying a plurality of security features embedded in the paper stock used to print officialballots. Additionally, if the security features include data (plain or encrypted), the voting unit 11 is capable of interpreting the data and verifying it. Further, if the security features include pre-assigned security codes, the voting unit II, for example using its scanner 29, is able to verify that the security codes present are authorized for that election. The necessary sensors to detect these latent features are included in the voting unit 11. These sensors consist of, but are not limited to, the following: Ultra-violet LED and sensor, Infra-red LED and sensor, magnetic sensor and the necessary electronics and software in order to decode the detected signals. FIG. 5 illustrates an example of a ballot 1 before the ballot has been filled out by a voter where voting marks can consist of, for example, separated ends of an arrow 6 that the voter can connect to cast a vote for a particular candidate. The ballot 1 can be, for example, 4.25 inches or 8.5 inches wide and from 11 inches to 22 inches in length. In one embodiment illustrated in FIG. 5, the ballot registration marks 3 are solid black 0.25 inch squares located just inside of a 0.25 inch unprinted area, bounding all sides of the ballot 1. Where the ballot 1 is longer than 11 inches, additional registration marks are desirable. The ballot can also include 'write in' areas 7, a machinereadable barcode 2 and a human readable version 4 of the machine-readable barcode 2 printed below the machine readable barcode 2 As noted earlier, the voting unit 11 additionally is capable of detecting and verifying a plurality of security features embedded in the printer stock and/or printed on the ballot 1

10

DETAILED DESCRIPTION OF EMBODIMENTS Latent Security Features FIGS. 1 and 2 illustrate an example of a paper stock having security features included therein. FIG. 1 shows a plurality of security features 8 that are embedded in the paper stock used to print officialballots. Such security features 8 could include, but are not limited to: ultraviolet features; infra-red features; magnetic features; fluorescence features; visual ink features; and watermarks. It is known to incorporate similar features in, for example, paper currencies. The implementation of such features in paper stock used to print official ballots can include, but is not limited to: shapes, words; numbers; images; I-D and 2-D barcodes; codes; and barcodes that can include anyone of real data and encrypted real data. For example, if the embedded feature is magnetic (such as an embedded metallic layer) the embedded magnetic material can have a particular shape, including the shape of a number, letter or word, or could be in the form of a barcode. Further, the security features embedded in the paper stock can include pre-assigned security codes from a pre-assigned set of codes. On example of a pre-printed security code would be encoding an "expiration" date on the paper. Using a simple I-D barcode, a numeric expiration date can be encoded on the paper using any of the latent features previously described. The ballot tabulator system can then be configured to reject paper with an expired code. Another example would be to encode a unique code, again utilizing a simple 1-D barcode pattern that must match the code assigned to the tabulator. In addition to the security features being embedded in the ballot paper stock, FIG. 2 shows security features 8' that can be printed on the ballot during the process of printing the official ballots (that is, during the process of printing the election choice information, an example of which is shown in FIG. 5, on the ballot). Such security feature properties can include, but are not limited to: ultraviolet features; infra-red features; magnetic features; fluorescence features; and visual ink features. The implementation of such features printed on the official ballots could include, but are not limited to: shapes; words; numbers; images; and I-D and 2-D barcodes, Further, the numbers, codes, and barcodes can include any one of real data and encrypted real data. Furthermore, the security features printed on the ballot during the process of printing official ballots can include pre-assigned security codes from a pre-assigned set of codes as well as pre-assigned ballot serial numbers from a pre-assigned set of serial numbers. As shown in FIGS. 1 and 2, the ballot security features can consist of, for example, a 1 inch series of bars 8 or 8' that are repeated every 3.5 inches along the length of the ballot. In some examples, a supplier may be a licensed authorized supplier of secure paper stock for ballot printing. By only allowing licensed paper suppliers to control and restrict access to the paper stock, the paper is not available to someone trying to forge ballots. It is understood that a ballot is provided by printing election choice information shown in FIG. 5, for example, on the paper stock of FIG. 1 or FIG. 2.

15

20

25

30

35

40

45

50

55

60

65

US 8,910,865 B2 5

6

during the process of printing official ballots. In the case where the security features include data (plain or encrypted), the optical scanner 29 is capable of interpreting the data and verifying it. Further, if the security features printed on the ballot 1 include a pre-assigned ballot serial number, the optical scanner 29 will be able to verify that the serial number present is authorized for that election and has not already been processed. The features specifically mentioned above can include but are not limited to: Ultraviolet features-These are features that are invisible when viewed under normal white light but become visible when illuminated by Ultra-violet light sources. They can also be features that absorb ultra-violet light. Typically these are inks Infra-red features-These are features that are invisible when viewed under normal white light but become visible when illuminated by Infra-red light sources. They can also be features that absorb infra-red light. Typicallythese are inks Magnetic features-These are features that have specific magnetic properties. Typically they are strips of magnetic material embedded in the paper, however magnetic inks also are available. The magnetic properties can be simple such as a uniform magnetic property or complex, such as a strip of material that has varying magnetic intensities along it which can represent a pattern or data. Fluorescent Features-These are features that may be visible or invisible when viewed under normal white light and fluoresce with an expected intensity range when illuminated by certain frequencies of light. Typically these are inks Visual ink features-These are features that are visible under normal light. Watermarks-Watermarks are typically physical features which are imprinted into the paper, either by embedding the layers within the paper or by being embossed into the paper. They are typically visible in normal white light but can not be replicated by printing techniques. (Note: Watermarks can also be Ultra-violet, Infra-red, fluorescent or magnetic features) A number of security features can be used in conjunction with each other to further improve security and make the forging or copying of ballots even more difficult The voting unit 11 includes one or more scanners (detectors) that are capable of detecting and reading the expected security features on the ballot 1. Such detectors are known to be used in currency authenticating apparatus. The definition of which security features to look for will form part of the ballot definition for the voting unit 11 so that the security features can be varied between jurisdictions, elections,-and even precincts. That is, a set of security features can be assigned to the ballots of each precinct, jurisdiction, election, etc., and the members of the set can be changed for different precincts, jurisdictions and elections, etc. One example of a combined set of security features would be the existence of UV fluorescent features, alternating with Infra-red features pre-printed on the ballot. These features would be detected with both a UV sensitive and IR sensitive sensor on the voting unit 11. These could also be combined with a human detectable water-mark. This water-mark can also be detected and processed by the optical scanner provided in the voting unit 11. The security features described above can be used suchthat they are grouped into three basic groups: static; dynamic; and data. Almost all of the types of features (UV, IR, magnetic, etc.) could belong to any of the groupings, depending on the implementation of the specific security features.

The group of static features refers to the situation where the feature is placed in the paper stock and is looked for by the voting unit 11. These static features do not contain data and thus the security features solely consist of the presence (or absence) of the feature. Typically, static features can, for example, consist of a mark in a set position or area on the ballot such that the positioning of the static feature does not change. Typically,features are static because they are expensiveto alter.For example, embedding magnetic strips in paper stock is a relatively expensive process. Therefore, it is likely that such features will be incorporated in a large volume of stock at one time and not altered frequently, if at all. Other static features may be selected because of the particular process that is used to create them. For example, a simple ink (such as UV or IR) feature could be applied during the paper stock manufacture process via a roller or brush. Such an application is relatively difficult to alter so again would be applied to large batches. Watermarks are another example of a security feature that is normally static. Dynamic features refer to features that can be varied, either in position, size, shape or content. Typically,features that are relatively cheap and easy to vary will be used as dynamic security features. For example, a feature which is somehow printed onto the stock during the manufacture process, such as a secure ink feature (using UV or IR sensitive) is often a dynamic feature. As it is printed at the time of manufacture, the position, shape and other properties could be altered for different batches of paper stock. Therefore, the dynamic security features can easily be varied for different elections,jurisdictions, or even districts to provide added security and prevent counterfeiting of ballots. Further, the voting unit 11 can be programmed to detect the specific feature, shape and location expected for the given election and jurisdiction. Data features are a special group of dynamic features. They contain data that can be read and verified by, for example, the scanner 29 of voting unit 11. Typically, the data will be represented in a feature such as a I-D or 2-D bar code. While the data could be anything, it is preferably a security code that can be validated. This data can be easily varied for different elections, jurisdictions, or even districts. To further increase the security of the code, the data can be encrypted using a pre-agreed private-public key pair. Thus, even if a potential forger managed to create some paper with the necessary feature technology (for example UV ink) and could reproduce the type offeature (say a barcode), the forger would have to know the correct security code to represent for that election. If the codes are encrypted, a scheme can be utilized that would require the forger to also have the public and private keys generated by the jurisdiction. Printed features, such as those using ultra-violet, infra-red, fluorescent, or magnetic ink could also be applied to each ballot by the ballot printer (the printer used to print a ballot such as the ballot shown in FIG. 5). This represents a different type of security as the 'source' of the security feature is not controlled; however, the content is and can be varied at a much lower level of granularity. For example each ballot style could have a printed security feature that has an encrypted code representing the election and ballot style along with the precincts in which they are valid. These security features could then be detected and verified by the scanner 29 of the voting unit 11. This improvement gives a very fine level of control and security to the ballot authentication process. The security features may also be masked by each other. For example, a feature that is printed using normal visible ink could have a different UV or IR feature printed on top of it. Further, if paper stock and ballot printer features are com-

10

15

20

25

30

35

40

45

50

55

60

65

US 8,910,865 B2 7

8

bined, it becomes virtually impossible-and certainly prohibitively expensive-to try to copy or forge ballots. Ballot Layout Authentication This improvement also includes a suitable procedure for calculating a unique value for a given set of contests and candidates positioned on a ballot 1. This value is included in a printable format in the image used to print the physical paper ballots 1. While the value may, or may not be human readable, the value is machine readable by the scanner 29 of voting unit 11. When the scanner 29 is programmed for use with a given ballot 1 (that is, for a given election), the unique layout value is included in the ballot definition. During the processing of the physical ballots, the value imprinted on the paper ballot 1 is compared to the value associated with the ballot definition on the scanner 29. If the values do not match, the scanner 29 will reject the ballot without further processing, or otherwise mark the ballot (via printer 30) as invalid. One possible implementation will be a hash calculation of the various candidate IDs and the associated target locations on the ballot face. However,there are a plurality of methods that can be employed to create a unique signature of the candidates and positions associated with all of the targets on the ballot face. By encoding this value on the ballot 1 itself, and then calculating the value again based on the electronic ballot definition used by the scanner 29 to process the ballot 1, the system can ensure the processing will match the physical layout of the ballot 1. Each voting unit 11 is provided a "ballot definition" of each ballot face valid for the voting unit 11 which includes the candidate ID and location (in x.y coordinates relative to the registration mark) of each votable target on the page. The concatenated list of these data points can generate a unique value (Hash) using a standard hash algorithm (SHA-l, SHA256). Each unique ballot face will generate a unique hash value when computed using the candidate/target position information. Once a ballot is scanned and the voting unit 11 assigns the correct ballot definition based on the ballot identifier, the unique hash signature can be recalculated using the ballot definition in order to compare to the value encoded on the ballot. The hash value can also be pre-calculated when the ballot definitions are loaded onto the voting unit 11. Each ballot definition will include a calculated hash value. This value can then be compared to the value encoded on the scanned ballot. In some examples, the fundamental ballot definition could be changed in an Election Management System (EMS) after the physical ballots have been printed. The scanner can then be initialized with the modified ballot layout definition. The modifications in the EMS/electronic ballot definition could include swapping the candidate positions between two candidates on the ballot as a way of altering the vote totals for a given contest. To prohibit such an occurrence, the layout validation and authentication feature can be calculated during the production of the images used to print the ballots 1. This feature will be a unique encrypted or human readable feature that uniquely represents the position of the targets on the printed ballot 1 in addition to the candidate and contest information. This value will be printed on the ballot 1 in such a way that the scanner 29 can read this value and then compare it to the electronic definition of the ballot 1 to ensure that the values, and hence the ballot target layout, are identical. The foregoing description is considered as illustrative only of the principles of the improvements discussed above. The inventions described herein are not limited to specific examples provided herein.

What is claimed is: 1. A method of validating and authenticating a votermarked paper ballot, the method comprising: calculating a unique authentication value based on election information provided on the voter-marked paper ballot; printing the authentication value on the voter-marked paper ballot as an encrypted security code in a printed security feature, the authentication value encrypted using a private-public key pair; associating the authentication value with a scanner that is configuredto receive and scan ballots having been completed by voters; scanning each voter-completed ballot to obtain the encrypted security code from the voter-marked paper ballot; decrypting the encrypted security code using the privatepublic key pair to obtain the authentication value; comparing the authentication value obtained from the ballot with the authentication value associated with the scanner; and physically marking the voter-marked paper ballot as invalid when the ballot-obtained authentication value does not match the authentication value associated with the scanner. 2. The method of claim 1, wherein the calculating a unique authentication value comprises performing a hash calculation of one or more candidate identifications and associated target locations on the ballot face. 3. The method of claim 2, wherein the hash calculation is performed using a concatenated list of data points representing coordinates of the one or more candidate identifications and associated target locations. 4. A method for authenticating ballots used in an election having multiple precincts, the method comprising: providing a plurality of ballots on which election-choiceinformation is printed, the ballots having a plurality of security features, wherein the plurality of security features printed on the ballot during the process of printing officialballots include pre-assigned security codes from a pre-assigned set of codes and a pre-assigned ballot serial number from a pre-assigned set of serial numbers; providing, from among the plurality of ballots, a first set of ballots having a first set of the plurality of security features associated with each ballot, the first set of security features including at least a firstencrypted security code, the first encrypted security code comprising a first security code that is encrypted using a private-public key pair; assigning the first set of ballots to a first precinct; providing, from among the plurality of ballots, a second set of ballots having a second set of the plurality of security features associated with each ballot, the second set of security features being different from the first set of security features and including at least a second encrypted security code, the second encrypted security code comprising a second security code that is encrypted using the private-public key pair; assigning the second set of ballots to a second precinct that is different from the first precinct; determining, after a vote has been cast, whether a particular ballot has the first set of security features and first security code or the second set of security features and second security code and whether the particular ballot was cast in the first precinct or the second precinct; and physically marking the particular ballot as invalidwhen the particular ballot does not have the security features and security code from the precinct in which the particular ballot was cast.

10

15

20

25

30

35

40

45

50

55

60

65

US 8,910,865 B2 9

10

5. The method of claim 4, wherein the first security code comprises a hash calculation of one or more candidate identifications and associated target location(s) on a first ballot face of ballots of the first set of ballots, and wherein the second security code comprises a hash calculation of one or more candidate identifications and associated target locations on a second ballot face of ballots of the second set of ballots. 6. The method of claim 5, wherein the hash calculations are based on concatenated list of data points representing coordinates of the one or more candidate identifications and associated target locations of the associated ballot faces. 7. A voting unit, comprising: a memory; a processor in communication with the memory, the processor controlling operations of the voting unit to: scan a voter-marked paper ballot used in an election, the voter-marked paper ballot comprising a first and second security features, wherein the first security feature is embedded in the voter-marked paper ballot, and the second security feature includes an encrypted security code printed on the voter-marked paper ballot; verify the first security feature by utilizing an electronic sensor to detect the first embedded security feature;

decrypt the second security feature by processing the encrypted security code with a private-public key; and authenticate the voter-marked paper ballot by comparing the verified first security feature and the decrypted second security feature with a set of pre-assigned first and second security features stored in the memory of the voting unit, wherein the voter-marked paper ballot is physically marked as invalid when the first and second security features do not match the set of preassigned first and second security features stored in the memory. 8. The voting unit of claim 7, wherein the pre-assigned first and second security features are assigned to the voter-marked paper ballots of each precinct, jurisdiction, or election. 9. The voting unit of claim 7, wherein the first security feature comprises a static feature, wherein verifying the first security feature comprises detecting the presence or absence of the first security feature. 10. The voting unit of claim 7, wherein the second security feature comprises an encrypted barcode. 11. The voting unit of claim 7, wherein at least some of the first and second security features overlap each other on the ballot.

10

15

20

* * * * *

111111

(12)

United States Patent

(10)

Hoover et al.

(45)

* * *

1112005 Pouloset al. 2/2006 Bogaskyet al. 10/2010 Chung

* cited by examiner Primary Examiner - Nirav G Patel (74) Attorney, Agent, or Firm - Holland & Hart LLP (57) ABSTRACT Methods, systems, and devices are described for adjudicating votes made on voter-marked paper ballots. Voter-marked paper ballots may be scanned to obtain optical image data of the voter-marked paper ballots. The optical image may be analyzed to determine the votes contained in the ballot for tabulation purposes. One or more votes on the ballot may be identified as requiring adjudication by an election official. Adjudication information, according to various embodiments, is appended to the optical images of the voter-marked paper ballots such that the image of the ballot and the image of the adjudication information may be viewed in an optical image. The optical image may be stored in a file format that allows the ballot image and the appended adjudication information to be viewed using readily available image viewers.

May 11, 2012 Prior Publication Data Nov. 14,2013

(51) Int. Cl. G06K 9/00 (2006.01) (52) U.S. Cl. USPC 382/103 (58) Field of Classification Search None See application file for complete search history.

20 Claims, 16 Drawing Sheets

City of Anywhere General Municipal Election Janual)f11891 TI)\'Ole,complel&lynUlnlhebox(es)

:;

~~V~R

W!'l

....

IZ1'ilI

I)RoberlL

MAYOR BORDON 0

m.I 'ijJean

CABOT

615-a~

ea

CHAMPlAIN

COOl( 0 ERIKSSON 0

7)John

FRANKLIN

91Jake

t:?Z?I

lDIWllliamLM.

m.I 11)'lVilfrid

ea

IZ)JohnA.

f2'?2J m

1311.e$tetB. 14)Louis$. 15JPiel11E.

ea

~~O:

COUNCILLOR::

WASSEI'I!iB

DO

~

6~

ea

0

O'DANIELD~ I/ONESPIE 00

2JMmelautP. J)Claust

ea f
0 HUDSON 0

ea

ea

0

5) I Sltell

IZi2I 8)Hamy

ea

0

•

5=~~:~::~ tlMuesJ.

""~)JaequeSCA fZ'.ZI 4)Samueld

ea

Dnexttovourchcleejsj,lIkethIS:

Qlndidalel

for
JOLIET

ea

0

m

KING0 LAURIER0 Ma1:DONALD

0

PEARSON SAINT-lAURENT TRUDEAU

0 0 0

V605-a

ea

£::!! ~

Ii:

~ _~~ ~ ~ ~ - - ~ ~.". ~

lit II 0 I!I @I 0

E====~D:J ...._ m

705176

235/386

Premier Election Solutions, Premier Central Scan User's Guide, Revision4.0, Jul. 9, 2009,201 pgs.

(21) Appl. No.: 13/470,091

m

235/386

OTHER PUBLICATIONS

Subject to any disclaimer, the term of this patent is extended or adjusted under 35 U.S.C. 154(b) by 50 days.

US 2013/0301873 Al

Dec. 16,2014

References Cited

2005/0247783Al 2006/0041516Al 2010/0252628Al

(73) Assignee: Dominion Voting Systems, Inc., Denver, CO (US)

(22) Filed:

US 8,913,787 B2

U.S. PATENTDOCUMENTS

(75) Inventors: James Hoover, Outremont (CA); Justin Bales, Chula Vista, CA (US); Steven Bennett, Lorna Linda, CA (US); Eric Coomer, Broomfield, CO (US); Sean Dean, Toronto (CA); Geneice Mathews, Golden, CO (US); Benjamin Rice, Brighton, CO (US)

( * ) Notice:

Patent No.: Date of Patent:

(56)

(54) BALLOT ADJUDICATION IN VOTING SYSTEMS UTILIZING BALLOT IMAGES

(65)

1111111111111111111111111111111111111111111111111111111111111 US008913787B2

):00

ea

~ z~

ea

'"

ea

m m m m m m m m ~h20-a

RO:2109 r:ce~:f=~~~~0'~'O'O,O,O,O,O,O,O.O.O,O

rmce

of Councillor (800,0) (1,790), (0,0) ote fur: Miles J. MASSEY (Ist), Menelaus P. O'DANIEL (2nd)

Office ofAldennan: 2,812,2,823,788,774 ote for: OVER·VOTE djudicationSessiononBatch I, Ballot I,by aer-c Admin from Machine ADJUDICATIONI Office of Mayor 0110211891 12;59:31 PM Office of Mayor hoice 3 > Marginal Mark; Mark ADDED ADnJDICATED OIlO2l1S9112:S9:36PMConles[

'---

900

u.s. Patent

Dec. 16,2014

US 8,913,787 B2

Sheet 1 of 16

Central Server Computer System 120

i+---H

Optical Scan Ballot System

115 105 Adjudication System 110

"<; FIG. 1

100

u.s. Patent

Dec. 16,2014

US 8,913,787 B2

Sheet 2 of 16

220

~

~

~ ~ ~ ~ ~ ~ ~ ~ ~

City of Anywhere

IZiZl

IZiZl

IZ:ZI IZ:ZI IZ:ZI

To vote, completely fill in the box(es) BAllOTFOR THE OFFICE OF

MAYOR

Vote for 0Il1y ONE(1)candidate

3) Jacques

f2Z:J f2Z:J

4) Samuel de

IZiZl

6) Leif

5) James

7) John

CARTIER CHAMPLAIN COOK ERIKSSON FRANKLIN

8) Henry

HUDSON

9) Jake

JOLIET

f2Z:J

10) William L.M.

IZiZl

11)Wilfrid

IZiZl

12) JohnA.

IZiZl

13) Lester B.

KING LAURIER MacDONALD PEARSON

IZ:ZI

14) Louis S.

SAINT·LAU RENT

IZ:ZI

15) Pierre E.

TRUDEAU

IZiZl

o next to your choice(s), BAllOT FOR THE OFFICE OF

0 0 0 0

Candida!e1 00 Candidate 2 • 0 Candidate 3

0 0 0 0 0 0

0 0 0

o.

MASSEY

1) MilesJ.

O'DANIEL

2) Menelaus P.

VON ESPIE

3) Clause .$ "TI OJ

~ !;!;

'" -
:::I

ce

..:!:; r c: 0

'<

•

like this:

COUNCILLOR

& ONE (1) 2nd choice, as shown alright

IZiZl

f2Z:J f2Z:J

E2Zl

Vow lor ONE (1) 1stchoice

205

IZiZl

E2Zl

General Municipal Election January 1 1891

IZ:ZI IZ:ZI

.!::;jJ

:z:

.!::l <-

~ '" iii· => ",

=>

iii:

'"

~ m

E2Zl E2Zl E2Zl @ @

210

@ <::o-
g_'T1::C»

~. it

.;;;:

E2Zl

,~ 0 0 0 0

nCJ

:r m

'"m

@ @ @ @

...... f2Z]

!;!; ::0

0 en m

:z "T1

215

0

~

C) ;:0; "T1

m r-

0

0

0

;;tI

:z -< Gl 0

!;!;

!;!;

c: :z:

m ::0 -<

'"'r= C)

G)

0000

0 E2Zl

IZiZl

~

~ E2Zl

IZiZl

~

~ ~ ~ ~ ~ ~ ~ ~ ~

"<; 200

FIG. 2

u.s. Patent

Dec. 16,2014

US 8,913,787 B2

Sheet 3 of 16

~~~~~~~~~~ General Municipal Election January 11891 To vote, completely fill in the box(es) BALLOT FOR THE OFFICE OF

MAYOR

1) Robert L. 2) Jean 3) Jacques 4) Samuel de 5) James 6) leif 7)John

BORDON

0

CABOT

0

CARTIER ~

0 0 0 0 0 0

3) Clause

9) Jake

JOLIET KING lAURIER

.$

., " => =>

'<

$

0

VON ESPIE

.:::

..!:;!.

13) lester B. 14) Louis S. 15) Pierre E.

I

Q c:

"0

8' :x: ;0 m m

....

5:

'" ro

(J)

("")

'" m,or ,,0

B ...::>'" Cl.

0

Z ....G"l s:: 0 0m "0 n n ;>; z -rt 0 ;s: r PEARSON 0 m " 0 m c r- ::0 ::0 :z: 0 0 -< G) SAINT·lAU RENT 0 TRUDEAU 0 O~O~~O 10

MacDONALD

~ ~

;;;:: ::0

12) John A.

O'DANIEl

<- m .- :z: g, 2. '"=>=> -< -<'" ~'"
;;;::

0 0~ 0 0

MASSEY ~

1) MilesJ,

COOK

HUDSON

11) Wilfrid

Candidate1 DO Candidate2 • 0 Candidate3 O.

"i:i- '\.~'t::.

2) Menelaus P,

FRANKLIN

•

fZ23

COUNCILLOR

shownatrighl

0

8) Henry

10) William L.M.

BALLOT FOR THE OFFICE OF

CHAMPLAIN

ERIKSSON

10\-11

fZ23

onext to your chclcets), like this: Vote for ONE(1)1st choice & ONE(1)2ndchoice,as

Vote for onlyONE(1)candidate

fZ23

ORO

City of Anywhere

;s:

(") » z » (")

m

::0 ::0

'"

.,.,-<

°0 "';0

fZ23

m m m

m m m (2(lJ (2(lJ

fZ23

m m

m m m (2(lJ

» r C

m

;;0

s: » z

fZ23

m m fZ23

m m

'<;

FIG. 3

300

u.s. Patent

Dec. 16,2014

US 8,913,787 B2

Sheet 4 of 16

~445

Auxiliary Ballot Box

Tabulator Ballot Box

415

410

"-

FIG. 4

400

u.s. Patent

~ ~

~

To vote, completely fill in the box(es) BAllOT FOR THE OFFICE OF

MAYOR

Vote for onlyONE(1)candidate

1) Robert L. Z) Jean 3) Jacques

BORDON CABOT CARTIER

0 0

IZZ?J 8) Henry I.Z2J 9) Jake IZZ?J 10) William L.M.

HUDSON

KING

IZ2Zl

LAURIER

0

~

5) James

I.Z2J

6) Leif

ERIKSSON

I.Z2J

7)John

FRANKLIN

11)Wilfrid

CHAMPLAIN COOK

JOLIET

13) Lester B.

IZZ?J 14) Louis S.

~

15) Pierre E.

~ ~

r

Candidate 1 DO Candidate 2 • 0 Candidate 3 O.

.... t;. MASSEY Q'DANIEL VON ESPIE

3) Clause

,\~l;,

I2ZI @ @ @ @ @ @ @

E2Zl

~

0

CiZJ

0

~

E2Zl

0 0

E2Zl @

~ ~ ..!:::! ~ I: z <m::> ~ '" ~ ~ .:2 (D' ~ ::>

.$ $ "T1 ;;;:: OJ ::l ::J '<

;;s:

CD


~ ~ffi~

Sf c:

-0

S

~~ CiCJ mAl

-i ::l: Al

r-n

;;;:: 0 "'0

Z

A

"T1

;0

@

ea /505

E2Zl

::>

0 0enm 0 -
'" c.

CiZJ

@

... ....

m

;:0

12) John A.

•

COUNCILLOR

Vote for ONE(1)1stchoice & ONE(1)2nd choice, as shown at right

2) Menelaus P.

0 0 0 0 0 0 0

4) Samuel de

BALLOT FOR THE OFFICE OF

1)MilcsJ.

I2ZI

lo\_'( I

o next to your choice(s), like this:

~

~

ORO

General Municipal Election January 11891

I.Z2J

~ ~

~ ~ ~ ~ ~ ~ ~ ~ ~

City of Anywhere

~ ~ ~ ~ ~ ~ ~ ~ ~ ~

US 8,913,787 B2

Sheet 5 of 16

Dec. 16,2014

c; Z

0;

~ 0>

('")

» ;;JO ::0

'"

» r

I2ZI

0

@

;;0

@

m !S:

» z

@ @ @

DRO:2109 Office Of Mayor: 0,0,1,0,0,0,0,0,0,0,0,0,0,0,0 Vote for: Jacques CARTIER

/510

Office of Councillor (800,0) (1,790), (0,0) Vote for: Miles J. MASSEY (1st), Menelaus P. O'DANIEL (2nd) Office of Alderman: 2,812,2,823,788,774 Vote for: Jeanne MANCE, Nellie McCLUNG, Mary PICKFORD

"<; 500 FIG. 5

u.s. Patent

Dec. 16,2014

~

~

Sheet 6 of 16

US 8,913,787 B2

~ ~ ~ ~ ~ ~ ~ ~ ~

~

f:z;zj f:z;zj

GeneralMunicipal Election January 11891

f:z;zj f:z;zj

To vote, completely fill in the box(es)

f:z;zj f:z;zj f:z;zj

MAYOR

f:z;zj

Vote forOl'lIy ONE(1) candidate

f:z;zj

1) Robert L.

~

615

BALLOTFOR THEOFFICE OF

Z) Jean

BORDON CABOT

D D

o next to your choice(s), BALLOTFOR THEOFFICE OF

f:z;zj

2) Menelaus P,

f:z;zj

3) Clause

f:z;zj f:z;zj f:z;zj

7) John

11)Wilfrid

f:z;zj

12) John A, 13) Lester B.

f:z;zj

14) Louis S.

f:z;zj

1S)Pierre E.

f:z;zj

[

.£:; rc:

..!::!

'TI

Z

0

PEARSON

0 0

:z

D

~~O~~D

KING lAURIER

SAINT·LAURENT TRUDEAU

10

~

~

:s:: to :2, '"" - ..;;;: ;;;' "" => " ;s:

D

JOLIET

f:z;zj

~

VON ESPIE

MacDONALD

9) Jake

Candidate 1 oD Candidate 2 • D Candidate 3 O.

O'DANIEL

0 0 0 0

HUDSON

10) William L.M,

D

•

f2ZI

like this:

<-

m

'"

en m

""Q

C'i

..... :s:: G"l (")

C")

:s::

Z

(")

0

m ,...

0

m

.» c: :z

0

0

-<

G"l

"Tl

co

:s:: ;>J

m

»

f2ZI

(22;) @ (22;)

<0-<(0 sa. .,,::c:p rn e-

Q c:

"0

or-

....

.,,0 ."

°0 m""

f2ZI f2ZI

:i!

(22;)

m

m

""

@

""'

(22;)

0;(")

f2ZI

0 ~ 0 0

"a; .,

Z

f2ZI

(22;)

0-

;;0;; .."

f2ZI

0

Q

0

f2ZI

(22;)

0

:s:: :xl

f2ZI

"I,~1;,

MASSEY ~

'<

8) Henry

f:z;zj

f:z;zj

.$

FRANKLIN

f2ZI

..... ~

1) MilesJ,

~

lo\_ yi COUNCILLOR

Vote forONE(1) 1st choice & ONE(1)2nd choice,as shown at right

f:z;zj

~

f2ZI

ORO

City of Anywhere

'"

::0 ::0

........._

» r C

m ;;0 ~ » z:

v-

605

f2ZI f2ZI (22;) (22;) @

E'dm

~

~

~6 20

~ ~ ~ ~ ~ ~ ~ ~

DRO: 2109 Office Of Mayor: 0,0,1,0,0,0,0,0,0,0,0,0,0,0,0 Vote for: AMBIGUOUS pffice of Councillor (800,0) (1,790), (0,0) Ivote for: Miles J. MASSEY (1st), Menelaus p, O'DANIEL (2nd) pffice of Alderman: 2,812,2,823,788,774 Ivote for: OVER-VOTE

"<, FIG. 6

600

u.s. Patent

Dec. 16,2014

115-a

110-a

US 8,913,787 B2

Sheet 7 of 16

B

/ Adjudication Computer 1

705

Adjudication Computer 2

Adjudication ComputerN

• • •

710

715

"-

FIG. 7

700

u.s. Patent

Dec. 16,2014

US 8,913,787 B2

Sheet 8 of 16

115-b

Receiver Module

110-b

805 Memory

Adjudication Module

815

810

820

User Interface

825

"-

FIG. 8

800

u.s. Patent

US 8,913,787 B2

Sheet 9 of 16

Dec. 16,2014

~ fZiZl

fZiZl

Tovote, completely fill in the box(es)

fZiZl~==========~~==========~ fZiZl BALLOT FOR THE OFFICE OF

fZiZl 1lRobertL 2.l Jean

~

0

CABOl

0

0

fZiZl 7) John

FRANKLIN

0

HUDSON

0 0 0

fZiZl

9) Jake

fZiZl

10) William L.M.

IZ?Zl

11)Wilfrid

fZiZl

12)JohnA.

IZ?Zl

13) LesterB.

PEARSON

0 0 0

fZiZl 14)Louis S. fZiZl15) Pierre E. fZiZl 1L--fZiZl

SAINT·LAURENT

0

TRUDEAU

0

L_

KING LAURIER MacDONALD

3) Clause

MASSEY

O'DANIEL VON ESPIE

~

'I,().1::>

0

0~ 0 0

0

fZiZl8) Henry

JOLIET

o.

....~

1) MilesJ.

ERIKSSON

Candidate I 0 0 Candidate 2 • 0 Candidate 3

Votetor ONE(1)lstchoice & ONE (112nd choice, as shOYnl alright

2) MenelausP.

fZiZl

•

COUNCILLOR

OF

BORDON

:~

615-.::aJ-....-~ 6)Leif

Onextto yourchoice(s), like this: 8ALLOTFOR THE OFFICE

MAYOR

fZiZl~Vo_w_~_OO~~~O_NE~(~I)_~_nd_~ mJ

lo\_yl

General Municipal Election January 11891

fZiZl

fZiZl

ORO

City of Anywhere

fZiZl

/605-a

---110 ~

PRO: 2109 pffice Of Mayor: 0,0,1,0,0,0,0,0,0,0,0,0,0,0,0 Vote for: AMBIGUOUS

/610-a

pffice of Councillor (800,0) (1,790), (0,0) ~ote for: Miles J. MASSEY (l st), Menelaus P. O'DANIEL (2nd) pffice of Alderman: 2,812,2,823,788,774 ~ote for: OVER-VOTE Adjudication Session on Batch - 1, Ballot - 1, by User - Admin from Machine ADJUDICATION 1 Office of Mayor 01/0211891 12:59:31 PM Office of Mayor Choice 3 > Marginal Mark: Mark ADDED 01/0211891 12:59:36 PM Contest ADJUDICATED

FIG. 9

"-

900

u.s. Patent

~

US 8,913,787 B2

Sheet 10 of 16

Dec. 16,2014

~

~ ~ ~ ~ ~ ~ ~ ~ ~

IZZI IZ:ZI IZZI IZZI IZZI IZZI

To vote, completely fill in the box(es) BALLOT FOR THE OFFICE OF

1) Robert

I.BiZJ

'l Jean

IZZI

3) Jacques

L.

IZZI 4) Samuel de IZZI 5) James IZZI 6) Leif IZZI IZZI

7) John

BORDON CABOT

2) Menelaus P.

COOK

0 0 0 0

3) Clause

FRANKLIN

JOLIET

11) Wilfrid

KING

"T1

..;g '"

MASSEY O'DANIEL VON ESPIE

~ ~ ~«: .; .z m m c: 3 zs: '" => ~ ~ -e ~ => m

.. ..

.$

.!:!J :;;:

(')

;;;:

0 0

c:

'0

I2Zl

I2Zl I2Zl I2Zl I2Zl I2Zl

.....~ ",J> 0 ~ 0 ~ 0 0

I2Zl

0'

=llSl

no m",

IZZI

I2Zl I2Zl I2Zl I2Zl I2Zl

:i!

I2Zl

m m

I2Zl I2Zl

9 g

:;;: z

......

12) JohnA.

Gl

0

:;;: m ;0 -<

SAINT·LAU RENT

1005

:::I

0

:s:: n

(")

r c:

z

C>

9, c.

~

3:

(") » z » ;0 (")

m

::0

'" '"

» r

I2Zl

0

I2Zl

;0

I2Zl

m :5:

» z

I2Zl I2Zl

0 ~

I2Zl

;;0

TRUDEAU

102

Q

I2Zl

o.

cr~~~ nr mr0'

IZZI IZ:ZI

Candidate 1 00 Candidate 2 • 0 Candidate 3

Vote forONE(1) 1stchoice & ONE (1) 2nd choice, as shown at right

0

ERIKSSON

•

COUNCILLOR

CHAMPLAIN

9) Jake

IZZI

BALLOTFOR THE OFFICE OF

1) MilesJ.

HUDSON

10) William L.M.

0 0

o next to your choice(s), like this:

~

CARTIER

8) Henry

IZ:ZI IZZI

MAYOR

Voie forOl1iy ONE (1lcandidate

IZZI IZZI

IZZI

I 0\-11

General Municipal Election January 11891

IZZI

I2Zl

ORO

City of Anywhere

~

~ ~ ~ ~ ~ ~ ~ ~ ~

020

Of Mayor: 0,0,1,0,0,0,0,0,0,0,0,0,0,0,0 for: OVER-VOTE

1010

of Councillor (800,0) (1,790), (0,0) ote for: Miles 1. MASSEY (1st), Menelaus P. O'DANIEL (2nd)

1015

FIG. 10

"-

1000

u.s. Patent

Dec. 16,2014

US 8,913,787 B2

Sheet 11 of 16

115-c

Network Interface

120-a

Election Information Collection and Management Module

1110 .----..__....,

Memory

1115

1120

User Interface

1125

"-

FIG. 11

1100

~

00 •

-e

~ 1""'1" ~

a e ('t

~

.....

%."'; ..~

~~

<"':''''1'

1'''

<..:>.

1,;>,,;;-..,

N

o

..... ,j;o.

:.,$ <;.i!;!! ;$.<:::,:,

ill. $-Sf~~~l

.,s.~~-G~1t_ $t*~~{~l ~**t~~~'(t&

l.,,.I<::)t~:;:_·

-.

01 "\

V~"":I-t"-~(, ...~ f<>l"""-nMn·'u,~FV~(~j :,

'1-

,.'C;:'('c.""

;~.. ~

.1. ~..tk;t'<

v ,,L (,,~,~

t" s:•.~

~~n~tor rM~~o~l U~nator Cho;$.co 1

.~.

u.u. 1:} ....:_.;;..

~>~"',,-n.Cit:..;"},'t*

ftl .. ~~';!if t r ...~1w

;:-;~I-t..%Jr_;'''l_J.1'''

:--;-\'~r~~li.)

.s.

t-=i·"t~"'),i ~t"') N~.-~C:'~'i"");cJ

I)""

~~ ...

)

1ZJ.

:r ('t

{()4S%)

i ~-l;,r~~

('t

~

.....

N

t1v. Cho~c.

.........

e

-Lv,"",

f",1.:$",

'* ~~ ...

(O-':>2~

rJT.::.;::~~A.f,~r') VA lM~1

U.S. Repr.o

I,.J

:.fi

~

.1 v~'''~

d

rJ1 QO

1225

FIG. 12

"-

1200

~ ~ """"

':...:J

QO

-..l

eo N

u.s. Patent

Dec. 16,2014

Sheet 13 of 16

US 8,913,787 B2

o

) o ~

u.s. Patent

Dec. 16,2014

US 8,913,787 B2

Sheet 14 of 16

Receive optical image data comprising an optical image of a voter-marked paper ballot

1'\1405

! Identify one or more votes to be adjudicated on the votermarked paper ballot

r-,1410

t Receive adjudication information for the one or more votes

! Generate image data comprising the received adjudication information

'"

1415

["\.1420

1 Append the image data comprising the received adjudication information for the ballot to the optical image data

FIG. 14

-.

1425

~1400

u.s. Patent

Dec. 16,2014

Sheet 15 of 16

Receive optical image data comprising an optical image of a voter-marked paper ballot

r-,1505

1 Identify one or more votes to be adjudicated on the votermarked paper ballot

I'- 1510

! Provide an image of the ballot for review by an election official

t-, 1515

1 Receive adjudication information from the election official

t-,1520

! Generate image data comprising the received adjudication information

"'1

525

! Append the image data comprising the received adjudication information for the ballot to the optical image data

"'1

530

~1500

FIG. 15

US 8,913,787 B2

u.s. Patent

Dec. 16,2014

Receive optical image data comprising an optical image of a voter-marked paper ballot

Sheet 16 of 16

US 8,913,787 B2

"'1605

! Determine vote information for the voter-marked paper ballot "1610

Provide the ballot optical image file for review by a second election official

1\ 1640

Receive adjudication information from the second election official

1\ 1645

! Append the vote information in human-readable format into the ballot optical image tile 1\1615

1 Identify one or more votes to be adjudicated on the votermarked paper ballot

"'1620

Append image data comprising the received adjudication information from the second election official to the optical 1\ 1650 image data

! Provide the ballot optical image file for review by an election official

1\1625

! Receive adjudication information from the election official 1\1630

! Append image data comprising the received adjudication information for the ballot to the "'1635 optical image data

I ~1600

FIG. 16

US 8,913,787 B2 1

2

BALLOT ADJUDICATION IN VOTING SYSTEMS UTILIZING BALLOT IMAGES

adjudication by an election official. Adjudication information, according to various embodiments, is appended to the optical images of the voter-marked paper ballots such that the image of the ballot and the image of the adjudication information may be viewed in an optical image. The optical image may be stored in a file format that allows the ballot image and the appended adjudication information to be viewed using readily available image viewers. In one set of embodiments, methods for ballot adjudication of a voter-marked paper ballot are provided that include receiving optical image data comprising an optical image of a voter-marked paper ballot, identifying one or more votes recorded on the voter-marked paper ballot to be adjudicated, receiving adjudication information for the ballot, generating image data comprising the received adjudication information for the ballot, and appending the image data comprising the received adjudication information for the ballot to the optical image data. In other embodiments, a system for adjudication of votermarked paper ballots is provided. The systems, according to various embodiments, may comprise a receiver module configured to receive ballot optical image data comprising an optical image of a voter-marked paper ballot, and an adjudication module configuredto receive adjudication information for the ballot, generate image data comprising the received adjudication information for the ballot and append the image data for the ballot to the optical image data. The adjudication module may be configured to receive an indication that the ballot requires adjudication when a mark within a target area of the ballot is determined to be ambiguous. In some embodiments, the adjudication module is further configured to provide an image of the ballot for review by an election official and receive the adjudication information from the election official. In some embodiments, multiple election officials may adjudicate a ballot, with each adjudication having corresponding image data comprising the received adjudication information for the ballot that is appended to the image data for the ballot.

BACKGROUND The present invention relates to voting systems in general and, in particular, to methods and systems for adjudicating ballots utilizing ballot images. Electronic tabulation of voter-marked paper ballots has been used in elections for some time. Such systems may provide efficient vote tabulation if ballots are received without any errors, modifications, or write-in candidates included on the ballot. Electronic tabulation systems for voter-marked ballots may include optical scanners that scan the votermarked ballots. Such ballots may have a target area that is evaluatedby a computer that receives an image of the scanned ballot to determine if a vote is cast for a particular election, such as a bubble or square next to a candidate's name on a ballot. Electronic tabulation systems may also receive ballots generated from direct-recording electronic (DRE) voting machines, in which a voter may electronically enter votes without a voter-marked paper ballot. In some instances optical scan systems may not be able to determine with high confidencethat a particular vote has been cast. For example, a voter may partially fill in a target area next to a candidate name, may place a mark next to the target area rather than filling in the target area, and/or soil the ballot in some fashion (e.g., coffee stain). Furthermore, in many jurisdictions voter intent is used as a basis for evaluating and adjudicating ballots for purposes of tabulating votes. Thus, if a voter improperly completed a ballot with a mark outside of a target area, a review of the ballot may indicate that the voter intended to cast a certain vote, and thus the ballot should be properly tabulated to include the intended vote. Similarly,a voter may mark an area for a particular candidate, and afterward realize that they made an error or change their mind. The voter may cross out the vote and mark a different area for a different candidate, and make a note such as, for example, writing "not this one" next to the crossed-out mark. An electronic scanning and tabulation system may not tabulate such a vote properly, while a human review of the ballot may readily reveal that the voter did intend to vote for the different candidate. Such instances may consume a significant amount of resources at the polling place or later, when the votes recorded on the ballots are being tabulated. Furthermore, many jurisdictions have a number of voters that vote through a mail-in ballot, and in some elections the voting may be done exclusively through mail-in ballots. Such mail-in ballots may have an increased likelihood of votes requiring adjudication, as the voter is not able to simply ask for a new ballot if they improperly mark or otherwise soil their original ballot. Furthermore, voters may complete mail-in ballots at their home, which may increase the likelihood that a ballot will be soiled with food or drink stains, which may lead to incorrect vote tabulation by an optical scan tabulation system.

10

15

20

25

30

35

40

45

50

55

SUMMARY 60

Methods, systems, and devices are described for adjudicating votes made on voter-marked paper ballots. Voter-marked paper ballots may be optically scanned to obtain optical image data of the voter-marked paper ballots. The optical image data may be analyzed to determine the votes contained in the ballot for tabulation purposes. One or more votes on the ballot, or the ballot as a whole, may be identified as requiring

65

BRIEF DESCRIPTION OF THE DRAWINGS A further understanding of the nature and advantages of the present disclosure may be realized by reference to the following drawings. In the appended figures, similar components or features may have the same reference label. Further, various components of the same type may be distinguished by following the reference label by a dash and a second label that distinguishes among the similar components. If only the first reference label is used in the specification, the description is applicable to anyone of the similar components having the same first reference label irrespective of the second reference label. FIG. 1 is a block diagram of a ballot scanning, tabulation, and adjudication system including components configured according to various embodiments. FIG. 2 is an image of a paper ballot according to various embodiments. FIG. 3 is an image of a voter-marked paper ballot according to various embodiments. FIG. 4 is an illustration of an optical scan ballot system according to various embodiments. FIG. 5 is an image of a voter-marked paper ballot with an audit mark according to various embodiments. FIG. 6 is an image of a voter-marked paper ballot containing an ambiguous mark according to various embodiments. FIG. 7 is a block diagram of an adjudication system according to various embodiments.

US 8,913,787 B2 3

4

FIG. 8 is a block diagram of an adjudication computer according to various embodiments. FIG. 9 is an image of a voter-marked paper ballot containing an ambiguous mark, an audit mark, and an adjudication mark, according to various embodiments. FIG. 10 is an image of a soiled voter-marked paper ballot containing an audit mark and an adjudication mark according to various embodiments. FIG. 11 is a block diagram of a central server computer system according to various embodiments. FIG. 12 is a screen shot provided by a central server computer system according to various embodiments. FIG. 13 is another screen shot provided by a central server computer system according to various embodiments. FIG. 14 is a flow chart illustrating operational steps of ballot adjudication according to various embodiments. FIG. 15 is another flow chart illustrating operational steps of ballot adjudication according to various embodiments. FIG. 16 is another flow chart illustrating operational steps of ballot adjudication according to various embodiments.

simply did not tabulate the vote because the mark for the candidate did not register as a vote for the candidate. When auditing election results, officialsmay desire to view the original ballot, along with any adjudications of votes on the ballot. In many instances, the assessment of what constitutes a vote in favor of a particular candidate or a particular answer to a question is determined by subjective assessment. Thus, in order to maintain integrity of the election process, a record of ballot adjudications may be desired, such that a record of the originally tabulated vote and any adjudicated votes are clear. With reference now to FIG. 1, a block diagram of a ballot scanning, tabulation, and adjudication system 100 according to some embodiments is described. In the system 100 of FIG. 1, an optical scan ballot system 105 is configured to receive voter-marked paper ballots, scan the ballots, and provide an optical image of the voter marked paper ballots. The optical scan ballot system 105 may include any of several types of scanning equipment, and in an embodiment includes a feeding mechanism that receives ballots and feeds the ballots through a scanner and then into a ballot box. In some embodiments' the optical scan ballot system marks each scarmed ballot with one or more marks that may be used to identify the paper ballot. In other embodiments, the optical scan ballot system 105 evaluates marks on the ballots and determines votes made by the voter. These determined votes may be stored for tabulation to determine election outcome. In some embodiments, determined votes may be printed on the ballots by the optical scan ballot system 105 using an internal printer. According to a set of embodiments, an optical image of the ballot is obtained by the optical scan ballot system 105, and in some embodiments the vote outcomes determined by the optical scan ballot system 105 are included as an audit mark in the optical image. Such an optical image may then be viewed by other systems to display the optical image of the voter-marked paper ballot along with the audit mark provided by the optical scan ballot system 105. Examples of optical scan ballot systems 105 will be described in more detail below. The ballot scanning, tabulation, and adjudication system 100 of FIG. 1 may also include adjudication system 110. The adjudication system 110 is connected through a network 115 to the optical scan ballot system 105 and to a central server computer system 120. Adjudication system 110 may receive data containing the optical images of paper ballots and audit marks and the votes recognized by the optical scan ballot system, if provided, from the optical scan ballot system 105. In some cases, the adjudication system 110 may receive data containing the optical images of paper ballots and audit marks from an election tabulation manager located at the central server computer system 120. An election official using the adjudication system 110 may review optical images of the paper ballots, and, based on the review,make a determination as to whether the marks were properly counted as votes or whether changes should be made to properly record the votes on the paper ballot. In some embodiments, an audit mark is viewable along with the image of the paper ballot allowingthe election official to review the votes recognized by the optical scan ballot system and compare the votes to the marks on the paper ballot. The election official may make any necessary changes and enter the changes through the adjudication system 110, which then may append an image to the optical image of the ballot that includes the adjudication information. Such an appended image may be referred to as an adjudication mark. Adjudication system 110 and central server computer system 120 will be described in additional detail below.

10

15

20

DETAILED DESCRIPTION Methods, systems, and devices are described for adjudicating votes made on voter-marked paper ballots. Voter-marked paper ballots may be scanned to obtain optical image data of the voter-marked paper ballots. The optical image may be analyzed to determine the votes contained in the ballot for tabulation purposes. One or more votes on the ballot, or the ballot as a whole, may be identified as requiring adjudication by an election official. Adjudication information, according to various embodiments, is appended to the optical images of the voter-marked paper ballots such that the image of the ballot and the image of the adjudication information may be viewed in an optical image. The optical image may be stored in a file format that allows the ballot image and the appended adjudication information to be viewed using readily available image viewers. This description provides examples, and is not intended to limit the scope, applicability or configuration of the invention. Rather, the ensuing description will provide those skilled in the art with an enabling description for implementing embodiments of the invention. Various changes may be made in the function and arrangement of elements. Thus, various embodiments may omit, substitute, or add various procedures or components as appropriate. For instance, it should be appreciated that the methods may be performed in an order different than that described, and that various steps may be added, omitted or combined. Also, aspects and elements described with respect to certain embodiments may be combined in various other embodiments. It should also be appreciated that the following systems, methods, devices, and software may individually or collectively be components of a larger system, wherein other procedures may take precedence over or otherwise modify their application. It may be beneficial to provide the ability to audit the election results. In some instances, one or more votes on a voter-marked paper ballot may need to be adjudicated by an election official. Such adjudication may be used in order to establish the intent of the voter that marked the paper ballot. For example, an optical scan tabulation system may not record a vote for a particular candidate because a box or bubble next to the candidate's name was not completely filled in. An election official reviewing such a ballot may readily conclude that the voter intended to cast a vote for the particular candidate, and that the optical scan tabulation system

25

30

35

40

45

50

55

60

65

US 8,913,787 B2 5

6

With reference now to FIG. 2, an example of a paper ballot 200 is described. The ballot 200 includes a number of voter selection areas 205, 210, 215 that represent voter selections for different offices or ballot questions. In the example ballot 200, there is also an election official area 220, which may be used by an election official to validate the particular ballot. Such validation may take the form of a signature or initials in the election official area 220 by an election official,however, other types of authentication may also be used. The election official area 220, according to a set of embodiments, servesto mark the paper ballot 200 as having been officially issued. It will be readily understood that paper ballots such as ballot 200 may include pre-printed ballots and/or ballots printed locally on-demand by a local printer. The voter will generally take the validated blank paper ballot 200 to a private area (such as a desk with privacy barriers), and make a mark or marks (this can be done in numerous ways such as filling in a box or oval, etc) beside the chosen candidate(s) with a marking instrument. While this disclosure may generally refer to selection of a candidate or candidates, it will be readily understood that some votes relate to other questions such as referendum questions, ballot initiatives, and the like. The concepts described herein extend to votes obtained all matters whether candidates, weighted candidate votes, referendum questions and the like. FIG. 3 illustrates an example of a ballot 300 that has been properly completed by a voter.After the voter has marked the ballot 300, the voter may provide the ballot to be tabulated. In some situations, the voter may drop the completed ballot (e.g., ballot 300) into a ballot box that election officials then take for processing and tabulation. In some embodiments, the election officials may direct the voter to place the ballot into an optical scan ballot system, such as optical scan ballot system 105 illustrated of FIG. 1, that scans the ballot and performs processing to determine the votes on the ballot. FIG. 4 is an illustration of a ballot receiving system 400 according to some embodiments. In one set of embodiments, the ballot receiving system 400 includes an optical scan ballot system 105-a that includes a hybrid paper/electronic vote tabulator 405. The hybrid paper/electronic vote tabulator 405, also referred to as a tabulation unit, is coupled with tabulator ballot box 410. An optional auxiliary ballot box 415 is also illustrated in FIG. 4, and may receive ballots that are not provided to the optical scanballot system 105.As voters come into the polling location, they may be processed by an election official who determines voter eligibility (based on local election rules), and also determines the proper ballot for the voter.The voter may then mark the ballot and take the marked paper ballot to the tabulation unit 405. In some embodiments, the tabulation unit 405 includes a display 420 that instructs the voter to feed the ballot into the tabulation unit 405 through path 425. The ballot may then be fed through the tabulator unit 405 and an image of the ballot created using a digital scanning device 430. According to various embodiments, the scanning device 430 takes a high resolution optical scan of the ballot and moves the ballot into the tabulator ballot box 410 through opening 435. In one particular aspect of the invention, the tabulation unit 405 takes the resulting high resolution scanned image of the entire ballot, and saves a copy of this image to a non-volatile memory linked to the optical scan ballot system lOS-a. System 400 may also include, as illustrated in FIG. 4, earphones 440, and a pendant manual trigger 445, for use by voters that require such devices. The tabulation unit 405 of a set of embodiments includes a processing module that executes software code to analyze the optical image of the ballot to determine the votes recorded on the ballot. The tabulation unit 405, in some embodiments,

also randomly assigns a filename to each scarmed image to ensure that the order in which the ballots were scarmed remains private. The tabulation unit 405 may also send a copy of the images to the central server computer system 120 and/or the adjudication system 110 of FIG. 1 for image recognition. In embodiments where tabulation unit 405 performs image processing, a ballot processing application runs an image recognition routine that is applied to the digital image and enables the tabulation unit 405 to selectively recognize specific areas of each image and may analyze such specific areas as described in more detail below to define a series of processing results associated with the particular ballot. For example, the tabulation unit 405 may analyze the various security markings on the ballot to ensure it is a valid ballot, perform a pixel count check to verify that the election official area 20 was initialed by the election official, and perform a pixel count of each voter selection area on the ballot. In addition, as further examples of such processing results, depending on the pixel count of each marking box, the mark may be classified as a 'vote', a 'non-vote.' or an 'ambiguous mark.' These classifications may be based, for example, on the total pixel counts of the marking areas (e.g., areas 205, 210, and215 in the example of FIG. 2) of the ballot, and/or a pattern of pixels in the marking areas. The determination of a vote, non-vote, or ambiguous mark is made according to pixel levels defined by election officials at a given time prior to the election. According to some embodiments, election officials may define, in pixels, the minimum pixel count that is to be classified as a 'vote,' the maximum pixel count (if any) that is to be defined as a definite 'nonvote,' and a range of pixels in between those values that will constitute an 'ambiguous mark.' These pixel values are loaded on each tabulation unit 405. After calculating these pixel values for each marking area, the tabulation unit 405 may return the ballot to the voter with an appropriate error message if any errors are detected in the ballot, or feed the ballot to the tabulator ballot box. An error may be detected, for example, if one or more voting areas contain an ambiguous mark, if too many marking areas in one category were classified as votes resulting in an "over-vote," ifno marking areas in one or more category were classified as votes (resulting in a blank ballot for one or more elections) and/or no pixel count was recorded in the election official area. In some embodiments, the tabulation unit 405 may be programmed to allow a voter to verify the ballot in the case of over-voted or blank ballots, thus preserving the voter's right to cast an over-voted or blank ballot. In other embodiments, the optical scan ballot system 105-amay return a ballot to the voter when an ambiguous mark or over-vote is detected, allowing the voter to correct the ballot or obtain a replacement ballot. In some embodiments, once the determination of the total votes for a ballot has been made, the tabulation unit 405 appends a footer to the saved ballot image that contains processing results for that specific ballot. FIG. 5 illustrates an image 500 that includes an optical image of a voter-marked paper ballot 505 as shown in FIG. 3, along with an audit mark 510 that is appended to the ballot optical image. The audit mark 510, according to some embodiments, is included as an image in the same file that contains the optical image of the voter-marked paper ballot. In such a manner, if a user displays the optical image of the voter-marked paper ballot, the audit mark will also be displayed allowing the user to view the ballot and voter markings as well as information on how the tabulation unit registered the votes for the ballot. Such a system allows for auditing of election results in an efficient manner. It will be understood that appending an optical audit mark image to the ballot optical image is just one marmer in

10

15

20

25

30

35

40

45

50

55

60

65

US 8,913,787 B2 7

8

which the votes recorded for a ballot may be associated with the ballot. In other embodiments, the information from the tabulation unit may be stored in a separate database and associated with a particular ballot, or may be stored in the ballot optical image file as extra data or metadata that mayor may not be displayed along with an image of the ballot. The embodiment of FIG. 5 illustrates the audit mark 510 at a footer of the optical ballot image 505, although it will be readily understood that an audit mark image 510 may be located at different locations relative to the ballot image 505. The audit mark 510 may also be referred to as a "fingerprint" or a "vote stamp" that is attached or otherwise associated with each image. In embodiments where the audit mark 510 is appended as an image to the voter-marked paper ballot optical image 505, the image may be savedin a format that is widely viewable by a variety of different image viewers. For example, the optical image may be stores as a .jpg file, a .tiff'file, or a .pdf'file, to name just a few examples. Various different types of viewers may be used to display such files, allowing the review of election results that does not require specialized software. Furthermore, the inclusion of an audit mark with the ballot optical image allows for review of how an optical scanning system read the particular ballot through simply viewing the optical image file. With reference again to FIG. 4, the ballot receiving system 400 may also include components to enable voting by an individual with a challenge such as visual impairment, illiteracy, inability to read the language in which the ballot is written, physical impairment with only limited motor abilities, etc. In such cases, an election official may initialize, for example, an "audio ballot" which may be administered directly by the tabulation unit 405. When the election official initializes, or activates, the audio ballot the voter may put on earphones 440 and take hold of the handheld pendant that includes the manual trigger selector 445. According to various embodiments, an audio ballot may simply be an audio version of the ballot, which may be stored to a local memory of the tabulation unit 405. The voter negotiates through the ballot through the audio interface and makes candidate selections via the manual trigger 440. In one embodiment, each time the voter makes a selection, and upon completion of voting for each individual race, the tabulation unit 405 asks the voter to confirm the choices after it is audibly reviewed. Upon completion of the audio ballot, a final review may be read back to the voter, and the voter is asked one more time to confirm the selected choices. In one embodiment, the tabulation unit 405 includes a printer that allows printing of the voter's selections on the ballot, and an optical image thereof may also be generated. In other embodiments, an audio clip of the final voter review, along with a subsequent audible voter confirmation of the review,may be recorded and stored in the tabulation unit 405. In other embodiments, this set of voter selections stored electronically that do not have a paper record can be used to create an optical image file that presents the vote selections in the same or similar format as a marked ballot, which can be printed and scarmedin a similar fashion as other scarmedpaper ballots. In addition, once the voter is finished with this final review, the election official may provide the voter one last chance to cancel the ballot and start over. Once the final review is completed, the tabulation unit 405 may be used to cast the audio ballot. By doing so, the votes from that audio ballot are appended to the ongoing tally of the tabulation unit 405, an audio recording to the process may be saved to the memory, and optionally a printer associated with the tabulation unit may prints the vote on a paper ballot or print a summary of the audio vote onto a paper chit.

The ballot or chit then falls into the tabulator ballot box 410 in order to maintain secrecy, or may be printed in encrypted form for audio playback on either the tabulation unit 405 or external device. At the end of voting, in real time, or periodically throughout the voting process, the tabulation unit 405, according to various embodiments, maintains an ongoing tally for all the ballots that passed through that particular machine, and those results may be transmitted by various known methods and technologies to a central location such as central server computer system 120 of FIG. 1. The central server computer system 120 may include, for example, a central election database that stores results from a number of optical scan ballot systems 105-a located throughout the election jurisdiction. The various election results from the various optical scan ballot systems 105-a may be summed together, provided to election officials, and stored to a memory, according to various embodiments. As described above, while voting occurs or once the vote has been completed, the optical image data for each of the scauned ballots along with the appended audit mark if present, may be transferred to the central server computer system 120. Election officials may perform various management, auditing, and adjudication tasks. In some embodiments, to ensure that the tabulation units 405 have functioned correctly, and also to verify the integrity of the vote overall, auditing of the optical image data may be performed. This may be in the form of a random spot check, a structured check to gain a certain confidencerate in the tabulator integrity,or a complete check of every ballot cast. Additionally, adjudication may be needed to determine proper votes cast on particular ballots. In some embodiments, an election information collection and management module located at the central server computer system, as will be described in additional detail below, identifiesballots that require adjudication. Such ballots may be ballots that were identified by the optical scan ballot system 105 as having errors, provisional or challenge ballots, and/or ballots that are damaged and unable to be read, for example. Ballots identified as requiring adjudication may be provided to an election official who may confirm, correct, or appropriately change one or more votes recorded for the particular ballot based on their review of the ballot optical image. For example, FIG. 6 illustrates a ballot optical image 600 including an image of the voter-marked paper ballot 605 and an appended audit mark 605. In this particular example, the ballot includes two errors. The first error is an improperly filled in selection 615, where the voter simply marked a box with an X rather than fillingin the box. The optical analysis of this ballot by the optical scan ballot system 105 (or other system that performs optical analysis of ballots to determine votes), as indicated in the audit mark 610, identifiedmark 615 as AMBIGUOUS. This may result from the number of pixels for the particular ballot area being below the defined threshold for counting the mark as a vote and above the defined threshold for identifying the mark as ambiguous, as described above. The second error on the ballot 605 is an overvote 620 for the office of Alderman. An election official may review this optical image and readily discern that mark 615 indicates that the voter intended to vote for Jacques Cartier for the officeof Mayor, and that mark 620 is, in fact, an overvote.The election official may update the vote recorded for the ballot 605 to include a vote for Jacques Cartier and include no vote for the officeofAlderman because the intention of the voter is not able to be discerned. Depending on the adjudication strategy used, an adjudication committee, for example, may have any number of termi-

10

15

20

25

30

35

40

45

50

55

60

65

US 8,913,787 B2 9

10

nals accessing the ballot optical images. With reference now to FIG. 7, an adjudication architecture 700 is described for a set of embodiments. An adjudication system 110-a may be coupled with network 115-a, and may receive optical image data for ballots identified as requiring adjudication. The optical image data may be provided by a central system, such as central server computer system 120 of FIG. 1, or may be provided directly from a ballot scanner such as optical scan ballot system 105 of FIG. 1 or 4. The adjudication system 110-a in this example includes a number of adjudication computers 705 through 715, that may be used to review ballot optical images, review audit marks associated with the ballots, and make determinations on whether particular marks on ballots should be recorded as votes. With reference now to FIG. 8, an adjudication architecture 800 according to another set of embodiments is described. In FIG. 8, an adjudication system 11O-b is coupled with network 115-b, and may receive optical image data for ballots identified as requiring adjudication. The optical image data may be provided by a central system, such as central server computer system 120 of FIG. 1, or may be provided directly from a ballot scanner such as optical scan ballot system 105 of FIG. 1 or 4. The adjudication system 11O-b in this example may be an example of one of the adjudication computers 705 through 715 of FIG. 7, may be a stand alone computer, or may be a computer that is connected directly to an optical scan ballot system. Adjudication system 110-b may be used to review ballot optical images, review audit marks associated with the ballots, and make determinations on whether particular marks on ballots should be recorded as votes. In some embodiments, adjudication system 110-b includes a receiver module 805, an adjudication module 810, a memory 815 that includes software 820, and a user interface 825. The receiver module 805 may include, for example, network interface hardware to allow connection with and communication over network 115-b. Such network interface hardware may include wired or wireless network interface cards and components, as are well nnderstood in the art. The adjudication module 810 may receive ballots to be adjudicated through the receiver module and may provide optical images of the ballots, and any included audit mark or other audit information, to the user interface 825. The user interface may include a monitor to display images to the election official, as well as a keyboard, mouse, or other data input device, as are well known. An election officialaccessing the user interface 825 may review the provided optical image and make a determination on votes for the ballot. The election official may provide input to the user interface 825, that the adjudication module 810 receives and uses to generate an adjudication mark. The adjudication module 810 may generate an optical image of the adjudication mark and append the adjudication mark to the ballot optical image, along with other relevant adjudication information. The optical image data including the adjudication mark may then be provided to the receiver module 805 and transmitted, for example, back to the central server computer system.Adjudication information may include adjudication of votes for a ballot, as well as information related to the date and time of adjudication, and identification of the election officialthat performed adjudication, for example. Memory 815 may include random access memory (RAM) and read-only memory (ROM), and store computer-readable, computer-executable software code 820 containing instructions that are configured to, when executed (or when compiled and executed), cause the adjudication module 810 to perform various functions described herein (e.g., provide ballot optical images for review, receive adjudication informa-

tion, append an optical image of the adjudication information to the ballot optical image, etc.). The components of the adjudication system 110-b may, individually or collectively, be implemented with one or more Application Specific Integrated Circuits (ASICs) adapted to perform some or all of the applicable functions in hardware. Each of the noted modules may be a means for performing one or more functions related to operation of the adjudication system 110-b. With reference now to FIG. 9, an example of a ballot optical image 900 following adjudication is described. In this example, the ballot optical image 900 includes an image of voter marked paper ballot 605-a and audit mark 610-a as illustrated in FIG. 6. As described above, the voter marked paper ballot 605-a includes two errors related to marks 615-a and 620-a. An election official may review the marks, and make an adjudication that mark 615-a is, in fact, a vote for Jacques Cartier, which is choice 3 on the ballot. The election official inputs this adjudication into the user interface of the adjudication system 110, and the adjudication system 110 generates image data 905 that includes the adjudication information. In the example of FIG. 9, the image data 905 of the adjudication information is appended to the ballot optical image data below the audit mark 610-a. In embodiments that do not include an audit mark image 610-a, image data 905 for the adjudication information may simply be appended to the ballot optical image data below the ballot optical image 605a, or at another suitable location adjacent the optical image of the ballot 605-a and/or audit mark 610-a. The adjudication information in image data 905, in this example, includes an identification of the user, namely the election official that performed the adjudication, an identification of the particular ballot question or office that was adjudicated, the change in the vote that was adjudicated, and a reason. In the example image data 905, it is noted that choice 3 was adjudicated (corresponding to Jacques Cartier) and that a marginal mark, corresponding to the ambiguous mark for that candidate, was added as a vote for the candidate. In such a manner, another user, election official, candidate representative, or observer may view the optical image and readily discern the image of the ballot as it was cast by the voter, the result of the initial optical scan of the ballot, and how the ballot was adjudicated. This can provide enhanced transparency and confidence in election results. With reference now to FIG. 10, an optical image 1000 of another ballot with audit and adjudication marks is described. In this example, a ballot optical image 1005, audit mark 1010, and adjudication mark 1015 are provided in optical image data. In this example, the optical scan ballot system that generated the audit mark 1010 identified two races as having over-votes. One resulting from mark 1020, and another resulting from a soiled area 1025 on the ballot 1005. Such soiled areas 1025 may be, for example, a coffee stain, food stain, or other type of soil on the ballot. Soiled ballots may be enconntered with increased frequency, for example, in mailin ballots. In this example, an election officialreadily discerns that mark 1020 is in fact an error on the ballot 1005, and that mark 1025, which resulted in the over-vote indication is a stray mark and not a vote. In this case, the election official adjudicated ballot 1005 to remove the stray mark and record the vote for that particular election area of the ballot 1005 to be for choice 3, corresponding to Jaques Cartier. It will be readily nnderstood that the examples of FIGS. 9 and 10 are but two examples of common situations requiring adjudication. Other examples include votes for write-in candidates, overvoteswhere the voter also includes a note that a particular vote is not to be counted, marks that are next to a candidates name instead of in the marking area for the votes, etc. Addi-

10

15

20

25

30

35

40

45

50

55

60

65

US 8,913,787 B2 11

12

tionally, multiple adjudications may be performed for any particular ballot, with each adjudication appended to the ballot optical image separately from other adjudications. In such a manner, an entire chain of adjudications may be viewed in the single optical image file, allowing a userto view the initial read of the ballot and each adjudication for the ballot. With reference now to FIG. 11, a system 1100 is described that includes central server computer system 120-a connected with network 115-c, according to a set of embodiments. The central server computer system 120-a may include a network interface 1105, an election information collection and management module 1110, memory 1115 that includes software 1120 stored therein, and a user interface 1125. The central server computer system 120-amay communicate with optical scan ballot system(s) and adjudication system(s), such as systems 105 and 110 of FIGS. 1, 4, and 7-8, through the network 115-c in order to receive and send information to such systems. The network interface 1105 may include, for example, commonly used network interface hardware to allow connection with and commnnication overnetwork 115c. Such network interface hardware may include wired or wireless network interface cards and components, as are well understood in the art. The election information collection and management module 1110 may receive optical image data from optical scan ballot systems and store the data in memory 1115. Memory 1115 may include a database that is located locally and/or remotely from the central server computer system 120-a. Furthermore, central server computer system 120-a itself may include a single computer, or may include multiple computers which may be located remotely from one another. In any event, the election information collection and management module 1110 may receive optical image data and use this data to determine cumulative vote connts for one or more elections, and to perform election management tasks such as assigning ballots identified as having errors for adjudication, and assigning ballots to be audited according auditing procedures for the jurisdiction to verify the optical scan ballot systems appear to be properly tabulating votes. In one embodiment, the election information collection and management module 1110 receives optical image data for each scanned ballot, along with audit mark information for each ballot, and maintains appropriate vote connts based on the data. If the audit mark information for a ballot indicates one or more errors for the ballot, the election information collection and management module 1110 may assign the ballot to one or more election officialsfor adjudication. The election officials may adjudicate the ballots using an adjudication system 110 such as described with respect to FIGS. 1, and 7-8. The election information collection and management module 1110 may receive the adjudicated ballot information back from adjudication systems and update the vote connts for the various elections based on the adjudication information. The election information collection and management module 1110 may then store the optical image data, including the ballot optical image and the adjudication information, in memory 1115 for future retrieval as needed. The user interface 1125 may include a monitor to display images to the election management officials, as well as a keyboard, mouse, or other data input device as are well known. An election official accessing the user interface 1125 may review optical images including one or more of audit mark information and adjudication information, as necessary. The election official may also use the user interface 1125 to monitor the status of ballots that have been assigned for adjudication and that have been received back at the election information collection and management module 1110 fol-

lowing adjudication. The user interface 1125 may also provide statistical information for use by election officials, as well as a number of other types of information for efficient election management. Memory 1115 may include random access memory (RAM) and read-only memory (ROM), and store computerreadable, computer-executable software code 1120 containing instructions that are configured to, when executed (or when compiled and executed), cause the election information collection and management module 1110 to perform various functions described herein (e.g., receive ballot optical images, identify images that require adjudication, receive adjudication information, maintain cumulative vote counts, etc.). The components of the central server computer system 120-a may, individually or collectively,be implemented with one or more Application Specific Integrated Circuits (ASICs) adapted to perform some or all of the applicable fnnctions in hardware. Each of the noted modules may be a means for performing one or more functions related to operation of the central server computer system 120-a. In one embodiment, the election information collection and management module 1110 executes software that allows for management of the election process, including assigning ballots that require adjudication to one or more election officials. FIG. 12 illustrates a screen shot 1200 that may be provided to an election official operating an adjudication system, such as adjudication system 110 of FIGS. 1, and 7-8. Variousareas of the screenprovide information to the election official, such as an indication of ballots pending 1205, that indicates how many ballots are pending for adjudication by the election official.The election officialmay select a ballot to adjudicate, and is provided with an indication 1210 of contents of the ballot that are awaiting adjudication. Different races for the particular election are listed at 1215, with an area 1220 provided to indicate the particular race or question that requires adjudication. The election official may view an image of the ballot, an image of an audit mark for the ballot, and/or an image of a prior adjudication mark in area 1225. The election official may adjudicate the ballot by selecting a candidate that is determined to have been selected by the voter and marking a check box for the candidate, illustrated in this embodiment at 1220. Followingthe completion of adjudication the softwaremay update the information provided to the election official,illustrated for one example in the screen shot 1300 of FIG. 13. In this example, following the adjudication of the ballot of the example of FIG. 12, the indication of ballots pending 1305, is updated to indicate how many ballots remain pending for adjudication by the election official. The indication 1310 of contents of the ballot awaiting adjudication is also updated, to indicate, in this example, that no other areas of the ballot require adjudication. Different races for the particular election are listed at 1315, which may be expanded to view the recorded vote on the ballot at area 1320. The election official may view an image of the adjudication mark that they just generated, an image of the ballot, an image of an audit mark for the ballot, and/or an image of a prior adjudication mark in area 1325. In such a manner, ballots may be reviewed and adjudicated in a transparent and efficient manner, and that allows for efficientand effectivemanagement through central server computer system 120. With reference now to FIG. 14, a method 1400 for adjudicating ballots is described. The method 1400 may, for example, be performed by an adjudication system, a central server computer system, or optical scan ballot system of FIGS. 1, 4, 7-8, and 11, or using any combination of the devices described for these figures. Initially, at block 1405,

10

15

20

25

30

35

40

45

50

55

60

65

US 8,913,787 B2 13

14

optical image data comprising an optical image of a votermarked paper ballot is received. As discussed above, optical image data may be received from an optical scan ballot system that scans a voter-marked paper ballot, for example. One or more votes to be adjudicated on the voter-marked paper ballot are identified, according to block 1410. Suchvotes may be identified, for example, as votes that have ambiguous marks on the voter-marked paper ballot, elections or races in which no vote is entered (an 'undervote'), elections or races where too many candidates have been voted for (an 'overvote'), and/or the presence of a write-in candidate. Such votes also may be identified as a result of a damaged or misread ballot where no vote selections can be determined. In some embodiments, votes may be identified as a result of a ballot as a whole that is to be adjudicated. A ballot as a whole may be identified for adjudication in a number of situations, such as misread or damaged ballots as mentioned above, provisional ballots, absentee or mailed-in ballots that may have different formats, etc. Votes requiring adjudication may be identified, in some examples, by a central server computer system, with the optical image data and identified votes for adjudication provided to an adjudication system over a network. In some embodiments, votes requiring adjudication may be identified by an optical scan ballot system, with adjudication of the votes also performed with an adjudication system that is located with or near such an optical scanballot system. In still other embodiments, areas of multiple ballots may be identified to be adjudicated, such as when a significant number of ballots are identified as not having a vote for a particular race, for example, which may indicate that a system may not be properly reading the ballots. With continuing reference to FIG. 14, adjudication information for the one or more votes is received, as indicated at block 1415. Such adjudication information may include, for example, an identification of the election officialthat is adjudicating the ballot, a date and time of adjudication, information on how the one or more votes have been adjudicated, and/or reasons for the adjudication (e.g., ambiguous mark, stray mark, etc.). Adjudication information may also include, in some embodiments, identification of a computer being used for adjudication and a location of the computer.At block 1420, image data is generated that comprises the received adjudication information. Such image data may include all, or a subset of the adjudication information, and in some examples, non-image data may be included with the image data that includes all or a subset of the adjudication information. Finally, at block 1425, the image data comprising the received adjudication information for the ballot is appended to the optical image data. The optical image data then includes data that comprises an optical image of the voter-marked paper ballot and an optical image of the adjudication information that may be displayed together when the optical image data is displayed. In some embodiments, the optical image data is in a format that is widely known and used, such as, for example, .jpg, .tiff, .bmp, or .pdf format. Having optical image data in such a format allows the ballot optical image and the adjudication information optical image to be viewed on many different platforms and using any of a number of widely available viewers, rather than requiring a proprietary viewer to view both the ballot and adjudication information. Storing the optical image data and adjudication information in the same optical data file also allows viewing of all of the pertinent election information related to the ballot without the need to access a separate database or data store that may have some of the information related to a ballot, such as adjudication information.

With reference now to FIG. 15, another method 1500 for adjudicating ballots is described. The method 1500 may, for example, be performed by an adjudication system, a central server computer system, or optical scan ballot system of FIGS. 1, 4, 7-8, and 11, or using any combination of the devices described for these figures. Initially, at block 1505, optical image data comprising an optical image of a votermarked paper ballot is received. As discussed above, optical image data may be received from an optical scan ballot system that scans a voter-marked paper ballot, for example. One or more votes to be adjudicated on the voter-marked paper ballot are identified, according to block 1510. Such votes may be identified as described above, for example, as votes that have ambiguous marks, undervote situations, overvote situations, and/or the presence of a write-in candidate. Such votes also may be identified as a result of a damaged or misread ballot where no vote selections can be determined. In some embodiments, votes may be identified as a result of a ballot as a whole that is to be adjudicated, similarly as described above. Votes requiring adjudication may be identified, in some examples, by a central server computer system, or by an optical scan ballot system, with adjudication of the votes performed with an adjudication system that is located either locally or remotely. In other embodiments, areas of multiple ballots may be identified to be adjudicated, similarly as described above. With continuing reference to FIG. 15, an image of the ballot for review is provided to an election official, as indicated at block 1515. In some embodiments, adjudication is performed on an adjudication system that runs specialized software the communicates with corresponding software on a central server computer system. The software at the central server computer system, in such embodiments, communicates ballots that require adjudication to the adjudication system, along with an identification of the votes on the ballot that require adjudication. At block 1520, adjudication information is received from the election official. The software at the adjudication system, in an example, provides the ballot and information related to votes requiring adjudication an election official who then adjudicates the ballot based on a visual review of the ballot. Adjudication information may include, for example, an identification of the election official that is adjudicating the ballot, a date and time of adjudication, information on how the one or more votes have been adjudicated, and/or reasons for the adjudication (e.g., ambiguous mark, stray mark, etc.). Adjudication information may also include, in some embodiments, identification of a computer being used for adjudication and a location of the computer.At block 1525, image data is generated that comprises the received adjudication information. Such image data may include all, or a subset of the adjudication information, and in some examples, non-image data may be included with the image data that includes all or a subset of the adjudication information. Finally, at block 1530, the image data comprising the received adjudication information for the ballot is appended to the optical image data. The optical image data then includes data that comprises an optical image of the voter-marked paper ballot and an optical image of the adjudication information that may be displayed together when the optical image data is displayed. In some embodiments, as described above, the optical image data is in a format that is widely known and used, allowing the ballot optical image and the adjudication information optical image to be viewed on many different platforms and using any of a number of widely available viewers. Storing the optical image data and adjudication information in the same optical data file also allows viewing of all of the

10

15

20

25

30

35

40

45

50

55

60

65

US 8,913,787 B2 15

16

pertinent election information related to the ballot without the need to access a separate database or data store that may have some of the information related to a ballot, such as adjudication information. In some embodiments, one or more ballots may be adjudicated multiple times. For example, all ballots having write-in candidates may be adjudicated, with ballots having ambiguous marks, overvotes, undervotes, or other errors may be separately adjudicated. Thus, ballots having both a write-in candidate and one or more other errors may be adjudicated multiple times. In other examples, separate officials may adjudicate ballots independently, and in the event that the adjudications do not agree, ajudge may make a finalruling on how a vote is to be adjudicated. For example, representatives of each candidate on a ballot may provide separate opinions on how a vote is to be adjudicated, with non-matching opinions ruled on by an elections judge. With reference now to FIG. 16, a method 1600 for adjudicating ballots multiple times is described. The method 1600 may, for example, be performed by an adjudication system, a central server computer system, or optical scan ballot system of FIGS. 1, 4, 7-8, and 11, or using any combination of the devices described for these figures. Initially, at block 1605, optical image data comprising an optical image of a voter-marked paper ballot is received. As discussed above, optical image data may be received from an optical scan ballot system that scans a votermarked paper ballot, for example. At block 1610, vote information for the ballot is determined. Such vote information may be determined as described above, for example, by performing a pixel-based analysis of voting and non-voting areas on the ballot to identify votes cast by the ballot, as well as any errors on the ballot, such as stray marks, overvotes, undervotes, and ambiguous marks. The vote information is appended to the optical image of the ballot in a humanreadable format, according to block 1615. At block 1620, one or more votes to be adjudicated on the voter-marked paper ballot are identified. Such votes may be identified as described above, as one or more votes that have an identified error, for example. An image of the ballot for review is then provided to an election official, as indicated at block 1625. In some embodiments, adjudication is performed on an adjudication system such as described above that runs specialized software as part of an election management software suite that runs across a central server computer system and one or more adjudication systems. At block 1630, adjudication information is received from the election official. Adjudication information may include, for example, an identification of the election officialthat is adjudicating the ballot, a date and time of adjudication, information on how the one or more votes have been adjudicated, and/or reasons for the adjudication (e.g., ambiguous mark, stray mark, etc.). Adjudication information may also include, in some embodiments, identification of a computer being used for adjudication and a location of the computer.At block 1635, image data is appended to the optical image data that comprises the received adjudication information. Such image data may include all, or a subset of the adjudication information. Next, at block 1640, an image of the ballot for review is provided to a second election official. As mentioned above, the second election official may be a representative of a candidate, a representative of a political party, or a second election official focused on a particular type of adjudication such as adjudication of write-in votes. Adjudication may be performed on an adjudication system such as described above.At block 1645, adjudication information is received from the second election official. Adjudication information may include, for example, an identification of the second election

official, along with one or more other items of adjudication information such as described above. At block 1650, image data is appended to the optical image data that comprises the received adjudication information from the second election official. Such image data may include all, or a subset of the adjudication information. In some embodiments more that two election officials may adjudicate a ballot, in which case the operations of blocks 1640 through 1650 are repeated for each additional election official.Thus, the optical image data includes an optical image of the ballot, and adjudication information from each election official that adjudicated the ballot. In some embodiments, as described above, the optical image data is in a format that is widely known and used, allowing the ballot optical image and the adjudication information from each election official to adjudicate the ballot to be viewed on many different platforms and using any of a number of widely available viewers. Storing the optical image data and adjudication information in the same optical data file also allows viewing of all of the pertinent election information related to the ballot without the need to access a separate database or data store that may have some of the information related to a ballot, such as adjudication information. It should be noted that the methods, systems and devices discussed above are intended merely to be examples. It must be stressedthat various embodiments may omit, substitute, or add various procedures or components as appropriate. For instance, it should be appreciated that, in alternative embodiments, the methods may be performed in an order different from that described, and that various steps may be added, omitted or combined. Also, features described with respect to certain embodiments may be combined in various other embodiments. Different aspects and elements of the embodiments may be combined in a similar manner. Also, it should be emphasized that technology evolvesand, thus, many of the elements are exemplary in nature and should not be interpreted to limit the scope of the invention. Specific details are given in the description to provide a thorough understanding of the embodiments. However,it will be understood by one of ordinary skill in the art that the embodiments may be practiced without these specificdetails. For example, well-known circuits, processes, algorithms, structures, and techniques have been shown without unnecessary detail in order to avoid obscuring the embodiments. Also, it is noted that the embodiments may be described as a process which is depicted as a flow diagram or block diagram. Although each may describe the operations as a sequential process, many of the operations can be performed in parallel or concurrently. In addition, the order of the operations may be rearranged. A process may have additional steps not included in the figure. Moreover, as disclosed herein, the term "memory" or "memory unit" may represent one or more devices for storing data, including read-only memory (ROM), random access memory (RAM), magnetic RAM, core memory, magnetic disk storage mediums, optical storage mediums, flash memory devices or other computer-readable mediums for storing information. The term "computer-readable medium" includes, but is not limited to, portable or fixed storage devices, optical storage devices, wireless channels, a sim card, other smart cards, and various other mediums capable of storing, containing or carrying instructions or data. Furthermore, embodiments may be implemented by hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof. When implemented in software, firmware, middleware or micro-

10

15

20

25

30

35

40

45

50

55

60

65

US 8,913,787 B2 17

18

code, the program code or code segments to perform the necessary tasks may be stored in a computer-readable medium such as a storage medium. Processors may perform the necessary tasks. Having described several embodiments, it will be recognized by those of skill in the art that various modifications, alternative constructions, and equivalents may be used without departing from the spirit of the invention. For example, the above elements may merely be a component of a larger system, wherein other rules may take precedence over or otherwise modify the application of the invention. Also, a number of steps may be undertaken before, during, or after the above elements are considered. Accordingly, the above description should not be taken as limiting the scope of the invention.

9. The method of claim 1, wherein identifying one or more votes recorded on the voter-marked paper ballot to be adjudicated comprises: determining that one or more portions of multiple ballots include one or more votes to be adjudicated. 10. The method of claim 1, wherein identifying one or more votes recorded on the voter-marked paper ballot to be adjudicated comprises: inspecting the ballot image to determine if a write-in vote has been selected; and assigning the write-in selection to a candidate. 11. The method of claim 1, wherein identifying one or more votes recorded on the voter-marked paper ballot to be adjudicated comprises: receiving an indication that one or more votes recorded on the voter-marked paper ballot require adjudication. 12. A system for adjudication of voter-marked paper ballots, comprising: a receiver module configured to receive ballot optical image data comprising an optical image of a votermarked paper ballot and an audit mark indicating how a tabulation unit registered voters marked on the votermarked paper ballot; and an adjudication module configured to receive adjudication information from one or more election officials,wherein the adjudication information includes a determination whether a particular mark on the ballot should be recorded as a vote, generate image data comprising the received adjudication information for the ballot and append the image data for the ballot to the optical image data. 13. The system of claim 12, wherein the adjudication module is configured to receive an indication that the ballot requires adjudication when a mark within a target area of the ballot is determined to be ambiguous. 14. The system of claim 12, wherein the adjudication module is configured to receive an indication that the ballot as a whole requires adjudication. 15. The system of claim 12, wherein the adjudication information comprises one or more of: an identificationof an election officialperforming the adjudication; a record that one or more votes on the ballot have been adjudicated; or a date and time of adjudication. 16. The system of claim 12, wherein the image data comprising the received adjudication information for the ballot is inserted as text in a margin of the optical image of the ballot. 17. A system for adjudication of voter-marked paper ballots, comprising: means for receiving optical image data comprising an optical image of a voter-marked paper ballot and an audit mark indicating how a tabulation unit registered votes marked on the voter-marked paper ballot; means for identifying one or more votes recorded on the voter-marked paper ballot to be adjudicated; means for providing an image of the ballot for review by one or more election officials; means for receiving adjudication information from the one or more election officials, wherein the adjudication information includes a determination whether a particular mark on the ballot should be recorded as a vote; means for generating image data comprising the received adjudication information; and means for appending the image data comprising the received adjudication information for the ballot to the optical image data.

What is claimed is: 1. A method comprising: receiving optical image data comprising an optical image of a voter-marked paper ballot and an audit mark indieating how a tabulation unit registered votes marked on the voter-marked paper ballot; identifying one or more votes to be adjudicated on the voter-marked paper ballot; providing an image of the ballot for review by one or more election officials; receiving adjudication information from the one or more election officials, wherein the adjudication information includes a determination whether a particular mark on the ballot should be recorded as a vote; generating image data comprising the received adjudication information; and appending the image data comprising the received adjudication information for the ballot to the optical image data. 2. The method of claim 1, wherein the adjudication information comprises: an identificationof an election officialperforming the adjudication; and a record that one or more votes on the ballot have been adjudicated. 3. The method of claim 2, wherein the adjudication information further comprises a date and time of adjudication. 4. The method of claim 1, wherein appending the image data comprising the received adjudication information for the ballot to the optical image data comprises: inserting text in a margin of the optical image of the ballot, the text including the adjudication information. 5. The method of claim 1, further comprising: displaying the optical image data including an image of the ballot and adjudication information appended to the image of the ballot. 6. The method of claim 1, wherein identifying one or more votes recorded on the voter-marked paper ballot to be adjudicated comprises: evaluating a target area on the ballot; and determining that a mark within the target area is ambiguous. 7. The method of claim 1, wherein identifying one or more votes recorded on the voter-marked paper ballot to be adjudicated comprises: determining that the ballot as a whole is to be adjudicated. 8. The method of claim 1, wherein identifying one or more votes recorded on the voter-marked paper ballot to be adjudicated comprises: determining that one or more portions of the ballot include a different number of votes than expected.

10

15

20

25

30

35

40

45

50

55

60

65

US 8,913,787 B2

19 18. The system of claim 17, further comprising: means for providing an image of the ballot for review by an election official; and means for receiving adjudication information from the election official. 19. The system of claim 17, wherein the adjudication information comprises one or more of: an identificationof an election officialperforming the adjudication; a record that one or more votes on the ballot have been adjudicated; or a date and time of adjudication. 20. The system of claim 17, wherein the means for incorporating the adjudication information into the ballot optical image file comprises: means for inserting text in a margin of the optical image of the ballot, the text including the adjudication information.

* * * * *

20

10

15

111111

(12)

1111111111111111111111111111111111111111111111111111111111111 US009202113B2

United States Patent

(10)

Hoover et al.

(45)

Patent No.: Date of Patent:

(52) U.S. Cl. CPC

(54) BALLOT ADJUDICATION IN VOTING SYSTEMS UTILIZING BALLOT IMAGES

(56)

References Cited U.S. PATENTDOCUMENTS

2005/0247783Al * 1112005 Pouloset al. 2006/0041516Al * 2/2006 Bogaskyet al. 2010/0252628Al * 10/2010 Chung

(73) Assignee: Dominion Voting Systems, Inc., Denver, CO (US)

* cited by examiner Primary Examiner - Nirav G Patel (74) Attorney, Agent, or Firm - Holland & Hart LLP

(21) Appl. No.: 14/539,684

(57) ABSTRACT Methods, systems, and devices are described for adjudicating votes made on voter-marked paper ballots. Voter-marked paper ballots may be scanned to obtain optical image data of the voter-marked paper ballots. The optical image may be analyzed to determine the votes contained in the ballot for tabulation purposes. One or more votes on the ballot may be identified as requiring adjudication by an election official. Adjudication information, according to various embodiments, is appended to the optical images of the voter-marked paper ballots such that the image of the ballot and the image of the adjudication information may be viewed in an optical image. The optical image may be stored in a file format that allows the ballot image and the appended adjudication information to be viewed using readily available image viewers.

Nov. 12, 2014 Prior Publication Data

US 2015/0071501 Al

Mar. 12,2015

Related U.S. Application Data (63) Continuation of application No. 13/470,091, filed on May 11, 2012, now Pat. No. 8,913,787. (51) Int. Cl. G06K9/00 G07C 13/00 G06T 11/60

(2006.01) (2006.01) (2006.01)

19 Claims, 16 Drawing Sheets mmnUiUUflI lQiiJO~ROy \..-

City of Anywhere

Gen8l7llMunicipalElection January11891 To""It,
ea aa

SAl..l.OTF(III ijiEOfRCE

9

\I)OoforonlyOfEll)"-

•• 1

MAYOR

..

aORooH'i]

1I!i2l(1S....,ld.

CHAMPlAIMO

,

=l)lIobonL

'" ea '" '" aa aa ea om

= om

J).b
~)hmiI' 6)1.M

,,-

,,Ip.....

101WilIiamLJ,j llIWiI~i~ 12}JDn.,1l

'"

lJ)l.ester8

'"

15)Pi.rreE

ea

ea

14Itoui<S.

iii

0 ERIKSSON 0 f~AII~LIN0 HUOSOH 0 JOLIET 0 ~TN"G 0

....

I!ZI

COUNCillOR::

SAl..l.OlFOR

:! ~~ =~)~-.. :

ccox

= ea

One
ea c. sa .. .;> ea MUSEy!!l 0 IZ:lI O'OANIELDIfI ea VONESPI~ DO aa

... Ct£~)lIld'O:e

C:!ncickot3

CABOrD CARTIER

ljllll .. J 2)Men$U'P l)CIaIIse

l,l.URIER 0 lI!.oOO"~lO PEARSON

0 0

SAI"T-lAURENTO TRUOE"uD

705176

235/386

Premier Election Solutions, Premier Central Scan User's Guide, Revision4.0, Jul. 9, 2009.*

Subject to any disclaimer, the term of this patent is extended or adjusted under 35 U.S.C. 154(b) by 0 days.

(65)

235/386

OTHER PUBLICATIONS

This patent is subject to a terminal disclaimer.

(22) Filed:

G06K 9/00442 (2013.01); G06T 11/60

(58) Field of Classification Search None See application file for complete search history.

(72) Inventors: James Hoover, Outremont (CA); Justin Bales, Chula Vista, CA (US); Steven Bennett, Lorna Linda, CA (US); Eric Coomer, Broomfield, CO (US); Sean Dean, Toronto (CA); Geneice Mathews, Golden, CO (US); Benjamin Rice, Brighton, CO (US)

Notice:

*Dec. 1,2015

(2013.01); G07C 13/00 (2013.01); G06T 2207110004 (2013.01)

(71) Applicant: Dominion Voting Systems, Inc., Denver, CO (US)

( *)

US 9,202,113 B2

DIiID~I!!!lO

0

~ fficeOfMayor:O,O,I,O,O,O,O,O,O,O,O,O,O,O,O,78D otefor:JacquesCARTIER fficeofCouncillor(800,O)(1,790),(O,O) ole for: Miles J. MASSEY (lSI), Menelaus p, O'DANIEL

(2nd)

fficeofAlderrnan:2,8l2,2,823,788,774 ole for: Jeanne MANCE, Nellie McCLUNG, Mary PICKFORD

'<;

500

u.s. Patent

Dec. 1,2015

US 9,202,113 B2

Sheet 1 of 16

Central Server Computer System 120

Optical Scan Ballot System 115 105 Adjudication System 110

"<, FIG. 1

100

u.s. Patent

Dec. 1,2015

US 9,202,113 B2

Sheet 2 of 16

220

m

m m ~ m ~ ~ m ~ m

City of Anywhere

!Z::::!

General Municipal Election

~

11891

•

~

COUNCILLOR

!Z::::! !Z::::! !Z::::! !Z::::!

DO

205 MASSEY

~ ~

4) Samuel de

~

6) Leil

~

7) John

~ ~ ~

ma ma ma ~

O'OANIEl

CHAMPLAIN

5) James ERIKSSON

!Z::::! c:: 'Q

8)

!Z::::!

6

:i

9) Jake

:;0

IT! IT!

1QlWilliam Lilli,

215

llil: 0

... z

MacDONALD

G"l 0 llil: m ;;0 -c

SAINT·LAURENT

~

!Z::::!

ma ma

!Z::::! !Z::::!

~

~ ~ ~ ~ ~ ~

m m m

"<;

FIG. 2

200

u.s. Patent

Dec. 1,2015

US 9,202,113 B2

Sheet 3 of 16

City of Anywhere Municipal Election 11891

• COUNCILLOR Candidate 1 0 0 CmdidaIe :2 • 0 Candidate 3

o.

O'OANIEL VON ESPIE ..f:.

f!!l:

~

.!:::!

iii'

'" '"::>

z l&:

.... ::>

'"

-....

O::~~~ flr

if c::;

-.::>

0 -i ::r::: ;:0

m~ °0 ~-i

no m;:o

m m n ::I

;;:::

""

0

MacDONALO

-0

0

...

:;:0;;

C')

0

!!!:

n

n ;;:::

a:: m

r c:

c -<

Cl

0

SAINT·LAU RENT

z ....

;:0

;:0

> z z n

~ !;

'"

m

"<; 300

FIG. 3

u.s. Patent

US 9,202,113 B2

Sheet 4 of 16

Dec. 1,2015

440

(5 ~445

Auxiliary

Tabulator

Ballot

Ballot

Box

Box

415

410

"<; 400 FIG. 4

u.s. Patent

Dec. 1,2015

US 9,202,113 B2

Sheet 5 of 16

~

~ ~

~ ~

~

~

~

~

~

fiZ:lJ

City of Anywhere

fiZ:lJ

~

Municipal Election 11891

tz:Z!

~

•

tz:Z! fiZ:lJ

~

COUNCILLOR

~

Candidate 1 OD Candidate 2 .D Candidate 3 O.

~ fiZ:lJ

~ ~ ~ ~ ~ ~

!SiZ'

e::zl

~

11 MilesJ.

fiZ:lJ

2) MenelausP

fiZ:lJ

3lCIause

~

~

6) Leif

~

1)Jonn

~ ~

Bl

O'OANIEL

e::zl e::zl

.;!:!

;z

!£.

i"=

e::zl

.:::::. ~._ i:

e::zl

"" :::> '" ::> '"

~ e::zl

9)Jake

505

WillillmLM.

:s:::

Wilfrid JannA.

'" '":>

-l e;')

!!l-

z

MacDONALD

0 !:

lesterB. louis S.

0

m

::0

SAINf.lAURENT

-<

Q.

e::zl

a:

e::zl

'"'"

e::zl e::zl e::zl

Pierre E.

e::zl

ffice Of Mayor: 0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,78D ote for: Jacques CARTIER

510

ffice of Councillor (800,0) (1,790), (0,0) ote for: Miles 1. MASSEY (1st), Menelaus P. O'DANIEL (2nd) ffice of Alderman: 2,812,2,823,788,774 ote for: Jeanne MANCE, Nellie McCLUNG, Mary PICKFORD

"<; 500 FIG. 5

u.s. Patent

Dec. 1,2015

US 9,202,113 B2

Sheet 6 of 16

City of Anywhere Municipal Election 1 1891

I2Zl

. !,,.,, ll,nJ

• COUNCILLOR Carldida!e 1 (] (] Qmdidall! 2 • (] Candidall!:I (].

I2Zl I2Zl !.:iZl I2Zl I2Zl I2Zl !.:iZl

1) Miles J. 2) Menelaus P.

31 Clause

IZZ] IZZ] IZZ]

f2Zl

615

!.:iZl !.:iZl I2Zl

605

IZZ]

f2Zl MacDONALD

f:lZl

PEARSON

f2Zl

SAINT·LAU RENT

f2Zl f:lZl

I:2Zl

620

ffice Of Mayor: 0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,780 ote for: AMBIGUOUS

610

ffice of Councillor (800,0) (1,790), (0,0) ote for: Miles J. MASSEY (Ist), Menelaus P. O'OANIEL (2nd) fficc of Alderman: 2,812,2,823,788,774 ote for: OVER-VOTE

"<, FIG. 6

600

u.s. Patent

Dec. 1,2015

US 9,202,113 B2

Sheet 7 of 16

115-a /

e -

-

110-a /

705

Adjudication

Adjudication

Computer 1

Computer 2

710

Adjudication

• ••

ComputerN 715

"<; FIG. 7

700

u.s. Patent

Dec. 1,2015

US 9,202,113 B2

Sheet 8 of 16

115-b

Receiver Module

1l0-b

805 Memory

Adjudication Module

815

810

820

User Interface

825

"<, FIG. 8

800

u.s. Patent

Dec. 1,2015

US 9,202,113 B2

Sheet 9 of 16

City of Anywhere General Municipal Election

lZ:2l lZ:2l

January 11891 Tovote,completely

~

in the boxles)

•

lZ:2l~==========~~============~ lZ:2l MAYOR lZ:2l lZ:2l "*loronlyONE(ll~ lZ:2l 1) Robert L

!Z':l:I

COUNCILLOR CMdidaIe 1 0 0 Carn:lidi!l9 2 • 0

~

Candidate 3

t
o.

!Z':l:I

SiiZl I 2) Jean

lZ:2l

!2Zl

1) MilesJ, O'DANIEl

t
VON ESPIE

!2Zl

-

!Z':l:I

2) Menelaus P 3) Clause $ Sl:

.!;:.

!:::!

~._

r; '"::> iii. -<'" ~ ~ ai' '" z

li=

"" '"

m

6°-40> .,,::1::1> mf""" -!il $lr ..,~

~

'" ...e'"

(';Cl m;o

10) William LM, Sl: 0

111Wilfrid 12) John A,

Z -I

MacDONALD

0 0 Sl:

13)lester g, 14l Louis S.

SAINT·lAURENT

@

&l

!Z':l:I

605-a

::>

Q.

is.

'" i!

» I

@

0

t
;::0

@

m

-<

;:

iZ:Z! 151Pierre E.

!Z':l:I

~ m

m

::0

t
!Z':l:I

-< %

lZ:2l lZ:2l lZ:2l lZ:2l

t
!Z':l:I

» z

iZ:Z! iZ:Z!

620-a

ffice Of Mayor: 0,0,1,0,0,O,O,O,0,0,0,0,0,0,0,78D ote for: AMBIGUOUS

610-a

ffice of Councillor (800,0) (1,790), (0,0) ote for: Miles 1MASSEY (Ist), Menelaus P. O'DANIEL (2nd) ffice of Alderman: 2,812,2,823,788,774 Vote for: OVER-VOTE djudication Session on Batch - I, Ballot - 1, by scr - Admin from Machine ADJUDICATION 1 Office of Mayor 01102/1891 12:59:31 PM Office of Mayor .hoice 3 > Marginal Mark: Mark ADDED 01/02/1891 12:59:36 PM Contest DJUDICATED

FIG. 9

905

"<;

900

u.s. Patent

Dec. 1,2015

US 9,202,113 B2

Sheet 10 of 16

m

m ~ m m ~ ~ m ~ ~

City of Anywhere General Municipal Election I.'222'l

11891

•

I.'222'l

Tovote.

COUNCILLOR

O'OANIEL

tz:::J tz:::J

m m m m m m tz:::J tz:::J

m m m m m m m m

l!!: 0

z -!

Cl 0

l!!: m ;:0

-<

1005

!ZZ] !ZZ]

~

m m m ~ m m m m

~

1020

Of Mayor: 0,0,1,0,0,0,0,0,0,0,0,0,0,0,0, 78D ote for: OVER-VOTE

1010

of Councillor (800,0) (1,790), (0,0) ote for: Miles J. MASSEY (1st), Menelaus P. O'DANIEL (2nd) of Alderman: 2,812,2,823,788,774 OVER-VOTE udication Session on Batch - I, Ballot - I, by - Admin from Machine ADJUDICATION 1 Office of Mayor 0110211891 12:59:31 PM Office of Mayor 3 > Stray Mark: Mark removed 0110211891 12:59:36 PM Contest TED

FIG. 10

1015

"-

1000

u.s. Patent

Dec. 1,2015

US 9,202,113 B2

Sheet 11 of 16

115-c

Network Interface

120-a

Election Information Collection and Management Module

1110

Memory

1115

1120

...---""'-----.

User Interface

1125

"<; 1100 FIG. 11

u.s. Patent

Dec. 1,2015

US 9,202,113 B2

Sheet 12 of 16

o o

N

.......

)

).

~\ e-

;:.

.. *-'

:~:

t:

"'S

...':-:-:.

-x

l

~:: "';.::.:

") ~..•;:

~:.

,-,:-

........ :.

{

:::::f

t

'~"

:.~:

f

::.~.

...::

f.::

~:~.:$'

,>

.~;.~ :t ::;':. <:t

...•~

{";. w'

~::

f

....... )

;~.. :' :..-' .....

...•... '...,.' :~.

't'

j::

.:.~

-r. . ....;:.~ -:

..::

,

.~.~.

t ;'"

:;.'

t

:} ..;:

::-:::

>.

.;....

,,«

.;.'

i:: f

:.;:

~:: .} ','

.~:

:>. .;.y: c .••-c

w

::::.

...••

.~:

.}

~:

":--::

...~ ~:

~'"

:~

..~

':',

"""

:::..•. »< :-..;. ,'''' ;;r->, otY' '::., *;.

••..:. »!. :« :«.

;:..••. :«'

:.~: ) .•.•~:

t

~1 -» :.~

:::

..•.~

u.s. Patent

Dec. 1,2015

Sheet 13 of 16

US 9,202,113 B2

o o

M

......

)

u.s. Patent

Dec. 1,2015

US 9,202,113 B2

Sheet 14 of 16

Receive optical image data comprising an optical image of a voter-marked paper ballot

1'\1405

! Identify one or more votes to be adjudicated on the votermarked paper ballot

I'-1410

1 Receive adjudication information for the one or more votes

I'-1415

1 Generate image data comprising the received adjudication information

-,1420

!

Append the image data comprising the received adjudication information for the ballot to the optical image data

FIG. 14

-,1425

~1400

u.s. Patent

Dec. 1,2015

US 9,202,113 B2

Sheet 15 of 16

Receive optical image data comprising an optical image of a voter-marked paper ballot

r-

1505

t Identify one or more votes to be adjudicated on the votermarked paper ballot

-.1510

1 Provide an image of the ballot for review by an election official

-,1515

1 Receive adjudication information from the election official

-, 1520

t Generate image data comprising the received adjudication information

-,1525

t Append the image data comprising the received adjudication information for the ballot to the optical image data

"\ 1530

~1500

FIG. 15

u.s. Patent

Dec. 1,2015

Receive optical image data comprising an optical image of a voter-marked paper ballot

Sheet 16 of 16

US 9,202,113 B2

"'1605

t Determine vote information for the voter-marked paper ballot

"1610

Provide the ballot optical image file for review by a second election official

1'\ 1640

! Append the vote information in human-readable format into the ballot optical image file "'1615

J Identify one or more votes to be adjudicated on the votermarked paper ballot

"'1620

Receive adjudication information from the second election official

Append image data comprising the received adjudication information from the second election official to the optical image data

"

1645

-.

1 Provide the ballot optical image file for review by an election official

"1625

1 Receive adjudication information from the election official

"'1630

1 Append image data comprising the received adjudication information for the ballot to the 1'\1635 optical image data

I

~1600

FIG. 16

1650

US 9,202,113 B2 1

2

BALLOT ADJUDICATION IN VOTING SYSTEMS UTILIZING BALLOT IMAGES

paper ballots may be optically scanned to obtain optical image data of the voter-marked paper ballots. The optical image data may be analyzed to determine the votes contained in the ballot for tabulation purposes. One or more votes on the ballot, or the ballot as a whole, may be identified as requiring adjudication by an election official. Adjudication information, according to various embodiments, is appended to the optical images of the voter-marked paper ballots such that the image of the ballot and the image of the adjudication information may be viewed in an optical image. The optical image may be stored in a file format that allows the ballot image and the appended adjudication information to be viewed using readily available image viewers. In one set of embodiments, methods for ballot adjudication of a voter-marked paper ballot are provided that include receiving optical image data comprising an optical image of a voter-marked paper ballot, identifying one or more votes recorded on the voter-marked paper ballot to be adjudicated, receiving adjudication information for the ballot, generating image data comprising the received adjudication information for the ballot, and appending the image data comprising the received adjudication information for the ballot to the optical image data. In other embodiments, a system for adjudication of votermarked paper ballots is provided. The systems, according to various embodiments, may comprise a receiver module configured to receive ballot optical image data comprising an optical image of a voter-marked paper ballot, and an adjudication module configuredto receive adjudication information for the ballot, generate image data comprising the received adjudication information for the ballot and append the image data for the ballot to the optical image data. The adjudication module may be configured to receive an indication that the ballot requires adjudication when a mark within a target area of the ballot is determined to be ambiguous. In some embodiments, the adjudication module is further configured to provide an image of the ballot for review by an election official and receive the adjudication information from the election official. In some embodiments, multiple election officials may adjudicate a ballot, with each adjudication having corresponding image data comprising the received adjudication information for the ballot that is appended to the image data for the ballot.

CROSS REFERENCES This application is a continuation of U.S. patent application Ser.No. 13/470,091, filed May 11,2012, entitled, "BALLOT ADJUDICATION IN VOTING SYSTEMS UTILIZING BALLOTIMAGES,"which is incorporated by reference in its entirety for all purposes. 10

BACKGROUND The present invention relates to voting systems in general and, in particular, to methods and systems for adjudicating ballots utilizing ballot images. Electronic tabulation of voter-marked paper ballots has been used in elections for some time. Such systems may provide efficient vote tabulation if ballots are received without any errors, modifications, or write-in candidates included on the ballot. Electronic tabulation systems for voter-marked ballots may include optical scanners that scan the votermarked ballots. Such ballots may have a target area that is evaluatedby a computer that receives an image of the scanned ballot to determine if a vote is cast for a particular election, such as a bubble or square next to a candidate's name on a ballot. Electronic tabulation systems may also receive ballots generated from direct-recording electronic (DRE) voting machines, in which a voter may electronically enter votes without a voter-marked paper ballot. In some instances optical scan systems may not be able to determine with high confidencethat a particular vote has been cast. For example, a voter may partially fill in a target area next to a candidate name, may place a mark next to the target area rather than filling in the target area, and/or soil the ballot in some fashion (e.g., coffee stain). Furthermore, in many jurisdictions voter intent is used as a basis for evaluating and adjudicating ballots for purposes of tabulating votes. Thus, if a voter improperly completed a ballot with a mark outside of a target area, a review of the ballot may indicate that the voter intended to cast a certain vote, and thus the ballot should be properly tabulated to include the intended vote. Similarly,a voter may mark an area for a particular candidate, and afterward realize that they made an error or change their mind. The voter may cross out the vote and mark a different area for a different candidate, and make a note such as, for example, writing "not this one" next to the crossed-out mark. An electronic scanning and tabulation system may not tabulate such a vote properly, while a human review of the ballot may readily reveal that the voter did intend to vote for the different candidate. Such instances may consume a significant amount of resources at the polling place or later, when the votes recorded on the ballots are being tabulated. Furthermore, many jurisdictions have a number of voters that vote through a mail-in ballot, and in some elections the voting may be done exclusively through mail-in ballots. Such mail-in ballots may have an increased likelihood of votes requiring adjudication, as the voter is not able to simply ask for a new ballot if they improperly mark or otherwise soil their original ballot. Furthermore, voters may complete mail-in ballots at their home, which may increase the likelihood that a ballot will be soiled with food or drink stains, which may lead to incorrect vote tabulation by an optical scan tabulation system.

15

20

25

30

35

40

45

BRIEF DESCRIPTION OF THE DRAWINGS

50

55

60

SUMMARY 65

Methods, systems, and devices are described for adjudicating votes made on voter-marked paper ballots. Voter-marked

A further understanding of the nature and advantages of the present disclosure may be realized by reference to the following drawings. In the appended figures, similar components or features may have the same reference label. Further, various components of the same type may be distinguished by following the reference label by a dash and a second label that distinguishes among the similar components. If only the first reference label is used in the specification, the description is applicable to anyone of the similar components having the same first reference label irrespective of the second reference label. FIG. 1is a block diagram of a ballot scanning, tabulation, and adjudication system including components configured according to various embodiments. FIG. 2 is an image of a paper ballot according to various embodiments. FIG. 3 is an image of a voter-marked paper ballot according to various embodiments. FIG. 4 is an illustration of an optical scan ballot system according to various embodiments.

US 9,202,113 B2

3

4

FIG. 5 is an image of a voter-marked paper ballot with an audit mark according to various embodiments. FIG. 6 is an image of a voter-marked paper ballot containing an ambiguous mark according to various embodiments. FIG. 7 is a block diagram of an adjudication system according to various embodiments. FIG. 8 is a block diagram of an adjudication computer according to various embodiments. FIG. 9 is an image of a voter-marked paper ballot containing an ambiguous mark, an audit mark, and an adjudication mark, according to various embodiments. FIG. 10 is an image of a soiled voter-marked paper ballot containing an audit mark and an adjudication mark according to various embodiments. FIG. 11 is a block diagram of a central server computer system according to various embodiments. FIG. 12 is a screen shot provided by a central server computer system according to various embodiments. FIG. 13 is another screen shot provided by a central server computer system according to various embodiments. FIG. 14 is a flow chart illustrating operational steps of ballot adjudication according to various embodiments. FIG. 15 is another flow chart illustrating operational steps of ballot adjudication according to various embodiments. FIG. 16 is another flow chart illustrating operational steps of ballot adjudication according to various embodiments.

For example, an optical scan tabulation system may not record a vote for a particular candidate because a box or bubble next to the candidate's name was not completely filled in. An election official reviewing such a ballot may readily conclude that the voter intended to cast a vote for the particu1ar candidate, and that the optical scan tabulation system simply did not tabulate the vote because the mark for the candidate did not register as a vote for the candidate. When auditing election results, officialsmay desire to view the original ballot, along with any adjudications of votes on the ballot. In many instances, the assessment of what constitutes a vote in favor of a particular candidate or a particular answer to a question is determined by subjective assessment. Thus, in order to maintain integrity of the election process, a record of ballot adjudications may be desired, such that a record of the originally tabulated vote and any adjudicated votes are clear. With reference now to FIG. 1, a block diagram of a ballot scanning, tabulation, and adjudication system 100 according to some embodiments is described. In the system 100 of FIG. 1, an optical scan ballot system 105 is configured to receive voter-marked paper ballots, scan the ballots, and provide an optical image of the voter marked paper ballots. The optical scan ballot system 105 may include any of several types of scanning equipment, and in an embodiment includes a feeding mechanism that receives ballots and feeds the ballots through a scanner and then into a ballot box. In some embodiments' the optical scan ballot system marks each scarmed ballot with one or more marks that may be used to identify the paper ballot. In other embodiments, the optical scan ballot system 105 evaluates marks on the ballots and determines votes made by the voter. These determined votes may be stored for tabulation to determine election outcome. In some embodiments, determined votes may be printed on the ballots by the optical scan ballot system 105 using an internal printer. According to a set of embodiments, an optical image of the ballot is obtained by the optical scan ballot system 105, and in some embodiments the vote outcomes determined by the optical scan ballot system 105 are included as an audit mark in the optical image. Such an optical image may then be viewed by other systems to display the optical image of the voter-marked paper ballot along with the audit mark provided by the optical scan ballot system 105. Examples of optical scan ballot systems 105 will be described in more detail below. The ballot scanning, tabulation, and adjudication system 100 of FIG. 1may also include adjudication system 110. The adjudication system 110 is connected through a network 115 to the optical scan ballot system 105 and to a central server computer system 120. Adjudication system 110 may receive data containing the optical images of paper ballots and audit marks and the votes recognized by the optical scan ballot system, if provided, from the optical scan ballot system 105. In some cases, the adjudication system 110 may receive data containing the optical images of paper ballots and audit marks from an election tabulation manager located at the central server computer system 120. An election official using the adjudication system 110 may review optical images of the paper ballots, and, based on the review,make a determination as to whether the marks were properly counted as votes or whether changes should be made to properly record the votes on the paper ballot. In some embodiments, an audit mark is viewable along with the image of the paper ballot allowingthe election official to review the votes recognized by the optical scan ballot system and compare the votes to the marks on the paper ballot. The election official may make any necessary changes and enter the changes through the adjudication sys-

10

15

20

25

DETAILED DESCRIPTION Methods, systems, and devices are described for adjudicating votes made on voter-marked paper ballots. Voter-marked paper ballots may be scanned to obtain optical image data of the voter-marked paper ballots. The optical image may be analyzed to determine the votes contained in the ballot for tabulation purposes. One or more votes on the ballot, or the ballot as a whole, may be identified as requiring adjudication by an election official. Adjudication information, according to various embodiments, is appended to the optical images of the voter-marked paper ballots such that the image of the ballot and the image of the adjudication information may be viewed in an optical image. The optical image may be stored in a file format that allows the ballot image and the appended adjudication information to be viewed using readily available image viewers. This description provides examples, and is not intended to limit the scope, applicability or configuration of the invention. Rather, the ensuing description will provide those skilled in the art with an enabling description for implementing embodiments of the invention. Various changes may be made in the function and arrangement of elements. Thus, various embodiments may omit, substitute, or add various procedures or components as appropriate. For instance, it should be appreciated that the methods may be performed in an order different than that described, and that various steps may be added, omitted or combined. Also, aspects and elements described with respect to certain embodiments may be combined in various other embodiments. It should also be appreciated that the following systems, methods, devices, and software may individually or collectively be components of a larger system, wherein other procedures may take precedence over or otherwise modify their application. It may be beneficial to provide the ability to audit the election results. In some instances, one or more votes on a voter-marked paper ballot may need to be adjudicated by an election official. Such adjudication may be used in order to establish the intent of the voter that marked the paper ballot.

30

35

40

45

50

55

60

65

US 9,202,113 B2 5

6

tem 110, which then may append an image to the optical image of the ballot that includes the adjudication information. Such an appended image may be referred to as an adjudication mark. Adjudication system 110 and central server computer system 120 will be described in additional detail below. With reference now to FIG. 2, an example of a paper ballot 200 is described. The ballot 200 includes a number of voter selection areas 205, 210, 215 that represent voter selections for different offices or ballot questions. In the example ballot 200, there is also an election official area 220, which may be used by an election official to validate the particular ballot. Such validation may take the form of a signature or initials in the election official area 220 by an election official,however, other types of authentication may also be used. The election official area 220, according to a set of embodiments, servesto mark the paper ballot 200 as having been officially issued. It will be readily understood that paper ballots such as ballot 200 may include pre-printed ballots and/or ballots printed locally on-demand by a local printer. The voter will generally take the validated blank paper ballot 200 to a private area (such as a desk with privacy barriers), and make a mark or marks (this can be done in numerous ways such as filling in a box or oval, etc) beside the chosen candidate(s) with a marking instrument. While this disclosure may generally refer to selection of a candidate or candidates, it will be readily understood that some votes relate to other questions such as referendum questions, ballot initiatives, and the like. The concepts described herein extend to votes obtained all matters whether candidates, weighted candidate votes, referendum questions and the like. FIG. 3 illustrates an example of a ballot 300 that has been properly completed by a voter.After the voter has marked the ballot 300, the voter may provide the ballot to be tabulated. In some situations, the voter may drop the completed ballot (e.g., ballot 300) into a ballot box that election officials then take for processing and tabulation. In some embodiments, the election officials may direct the voter to place the ballot into an optical scan ballot system, such as optical scan ballot system 105 illustrated of FIG. 1, that scans the ballot and performs processing to determine the votes on the ballot. FIG. 4 is an illustration of a ballot receiving system 400 according to some embodiments. In one set of embodiments, the ballot receiving system 400 includes an optical scan ballot system 105-a that includes a hybrid paper/electronic vote tabulator 405. The hybrid paper/electronic vote tabulator 405, also referred to as a tabulation unit, is coupled with tabulator ballot box 410. An optional auxiliary ballot box 415 is also illustrated in FIG. 4, and may receive ballots that are not provided to the optical scanballot system 105.As voters come into the polling location, they may be processed by an election official who determines voter eligibility (based on local election rules), and also determines the proper ballot for the voter.The voter may then mark the ballot and take the marked paper ballot to the tabulation unit 405. In some embodiments, the tabulation unit 405 includes a display 420 that instructs the voter to feed the ballot into the tabulation unit 405 through path 425. The ballot may then be fed through the tabulator unit 405 and an image of the ballot created using a digital scanning device 430. According to various embodiments, the scanning device 430 takes a high resolution optical scan of the ballot and moves the ballot into the tabulator ballot box 410 through opening 435. In one particular aspect of the invention, the tabulation unit 405 takes the resulting high resolution scanned image of the entire ballot, and saves a copy of this image to a non-volatile memory linked to the optical scan ballot system lOS-a. System 400 may also include, as illus-

trated in FIG. 4, earphones 440, and a pendant manual trigger 445, for use by voters that require such devices. The tabulation unit 405 of a set of embodiments includes a processing module that executes software code to analyze the optical image of the ballot to determine the votes recorded on the ballot. The tabulation unit 405, in some embodiments, also randomly assigns a filename to each scarmed image to ensure that the order in which the ballots were scarmed remains private. The tabulation unit 405 may also send a copy of the images to the central server computer system 120 and/or the adjudication system 110 of FIG. 1 for image recognition. In embodiments where tabulation unit 405 performs image processing, a ballot processing application runs an image recognition routine that is applied to the digital image and enables the tabulation unit 405 to selectively recognize specific areas of each image and may analyze such specific areas as described in more detail below to define a series of processing results associated with the particular ballot. For example, the tabulation unit 405 may analyze the various security markings on the ballot to ensure it is a valid ballot, perform a pixel count check to verify that the election official area 20 was initialed by the election official, and perform a pixel count of each voter selection area on the ballot. In addition, as further examples of such processing results, depending on the pixel count of each marking box, the mark may be classified as a 'vote', a 'non-vote.' or an 'ambiguous mark.' These classifications may be based, for example, on the total pixel counts of the marking areas (e.g., areas 205, 210, and215 in the example of FIG. 2) of the ballot, and/or a pattern of pixels in the marking areas. The determination of a vote, non-vote, or ambiguous mark is made according to pixel levels defined by election officials at a given time prior to the election. According to some embodiments, election officials may define, in pixels, the minimum pixel count that is to be classified as a 'vote,' the maximum pixel count (if any) that is to be defined as a definite 'nonvote,' and a range of pixels in between those values that will constitute an 'ambiguous mark.' These pixel values are loaded on each tabulation unit 405. After calculating these pixel values for each marking area, the tabulation unit 405 may return the ballot to the voter with an appropriate error message if any errors are detected in the ballot, or feed the ballot to the tabulator ballot box, An error may be detected, for example, if one or more voting areas contain an ambiguous mark, if too many marking areas in one category were classified as votes resulting in an "over-vote," ifno marking areas in one or more category were classified as votes (resulting in a blank ballot for one or more elections) and/or no pixel count was recorded in the election official area. In some embodiments, the tabulation unit 405 may be programmed to allow a voter to verify the ballot in the case of over-voted or blank ballots, thus preserving the voter's right to cast an over-voted or blank ballot. In other embodiments, the optical scan ballot system 105-amay return a ballot to the voter when an ambiguous mark or over-vote is detected, allowing the voter to correct the ballot or obtain a replacement ballot. In some embodiments, once the determination of the total votes for a ballot has been made, the tabulation unit 405 appends a footer to the saved ballot image that contains processing results for that specific ballot. FIG. 5 illustrates an image 500 that includes an optical image of a voter-marked paper ballot 505 as shown in FIG. 3, along with an audit mark 510 that is appended to the ballot optical image. The audit mark 510, according to some embodiments, is included as an image in the same file that contains the optical image of the voter-marked paper ballot. In such a manner, if a user displays the optical image of the voter-marked paper ballot, the audit

10

15

20

25

30

35

40

45

50

55

60

65

US 9,202,113 B2 7

8

mark will also be displayed allowing the user to view the ballot and voter markings as well as information on how the tabulation unit registered the votes for the ballot. Such a system allows for auditing of election results in an efficient manner. It will be understood that appending an optical audit mark image to the ballot optical image is just one manner in which the votes recorded for a ballot may be associated with the ballot. In other embodiments, the information from the tabulation unit may be stored in a separate database and associated with a particular ballot, or may be stored in the ballot optical image file as extra data or metadata that mayor may not be displayed along with an image of the ballot. The embodiment of FIG. 5 illustrates the audit mark 510 at a footer of the optical ballot image 505, although it will be readily understood that an audit mark image 510 may be located at different locations relative to the ballot image 505. The audit mark 510 may also be referred to as a "fingerprint" or a "vote stamp" that is attached or otherwise associated with each image. In embodiments where the audit mark 510 is appended as an image to the voter-marked paper ballot optical image 505, the image may be savedin a format that is widely viewable by a variety of different image viewers. For example, the optical image may be stores as a .jpg file, a .tiff'file, or a .pdf'file, to name just a few examples. Various different types of viewers may be used to display such files, allowing the review of election results that does not require specialized software. Furthermore, the inclusion of an audit mark with the ballot optical image allows for review of how an optical scanning system read the particular ballot through simply viewing the optical image file. With reference again to FIG. 4, the ballot receiving system 400 may also include components to enable voting by an individual with a challenge such as visual impairment, illiteracy, inability to read the language in which the ballot is written, physical impairment with only limited motor abilities, etc. In such cases, an election official may initialize, for example, an "audio ballot" which may be administered directly by the tabulation unit 405. When the election official initializes, or activates, the audio ballot the voter may put on earphones 440 and take hold of the handheld pendant that includes the manual trigger selector 445. According to various embodiments, an audio ballot may simply be an audio version of the ballot, which may be stored to a local memory of the tabulation unit 405. The voter negotiates through the ballot through the audio interface and makes candidate selections via the manual trigger 440. In one embodiment, each time the voter makes a selection, and upon completion of voting for each individual race, the tabulation unit 405 asks the voter to confirm the choices after it is audibly reviewed. Upon completion of the audio ballot, a final review may be read back to the voter, and the voter is asked one more time to confirm the selected choices. In one embodiment, the tabulation unit 405 includes a printer that allows printing of the voter's selections on the ballot, and an optical image thereof may also be generated. In other embodiments, an audio clip of the final voter review, along with a subsequent audible voter confirmation of the review,may be recorded and stored in the tabulation unit 405. In other embodiments, this set of voter selections stored electronically that do not have a paper record can be used to create an optical image file that presents the vote selections in the same or similar format as a marked ballot, which can be printed and scanned in a similar fashion as other scanned paper ballots. In addition, once the voter is finished with this final review, the election official may provide the voter one last chance to cancel the ballot and start over. Once the final review is completed, the tabulation unit

405 may be used to cast the audio ballot. By doing so, the votes from that audio ballot are appended to the ongoing tally of the tabulation unit 405, an audio recording to the process may be saved to the memory, and optionally a printer associated with the tabulation unit may prints the vote on a paper ballot or print a summary of the audio vote onto a paper chit. The ballot or chit then falls into the tabulator ballot box 410 in order to maintain secrecy, or may be printed in encrypted form for audio playback on either the tabulation unit 405 or external device. At the end of voting, in real time, or periodically throughout the voting process, the tabulation unit 405, according to various embodiments, maintains an ongoing tally for all the ballots that passed through that particular machine, and those results may be transmitted by various known methods and technologies to a central location such as central server computer system 120 of FIG. 1. The central server computer system 120 may include, for example, a central election database that stores results from a number of optical scan ballot systems 105-a located throughout the election jurisdiction. The various election results from the various optical scan ballot systems 105-a may be summed together, provided to election officials, and stored to a memory, according to various embodiments. As described above, while voting occurs or once the vote has been completed, the optical image data for each of the scanned ballots along with the appended audit mark if present, may be transferred to the central server computer system 120. Election officials may perform various management, auditing, and adjudication tasks. In some embodiments, to ensure that the tabulation units 405 have functioned correctly, and also to verify the integrity of the vote overall, auditing of the optical image data may be performed. This may be in the form of a random spot check, a structured check to gain a certain confidencerate in the tabulator integrity,or a complete check of every ballot cast. Additionally, adjudication may be needed to determine proper votes cast on particular ballots. In some embodiments, an election information collection and management module located at the central server computer system, as will be described in additional detail below, identifiesballots that require adjudication. Such ballots may be ballots that were identified by the optical scan ballot system 105 as having errors, provisional or challenge ballots, and/or ballots that are damaged and unable to be read, for example. Ballots identified as requiring adjudication may be provided to an election official who may confirm, correct, or appropriately change one or more votes recorded for the particular ballot based on their review of the ballot optical image. For example, FIG. 6 illustrates a ballot optical image 600 including an image of the voter-marked paper ballot 605 and an appended audit mark 605. In this particular example, the ballot includes two errors. The first error is an improperly filled in selection 615, where the voter simply marked a box with an X rather than fillingin the box. The optical analysis of this ballot by the optical scan ballot system 105 (or other system that performs optical analysis of ballots to determine votes), as indicated in the audit mark 610, identifiedmark 615 as AMBIGUOUS. This may result from the number of pixels for the particular ballot area being below the defined threshold for counting the mark as a vote and above the defined threshold for identifying the mark as ambiguous, as described above. The second error on the ballot 605 is an overvote 620 for the office of Alderman. An election official may review this optical image and readily discern that mark 615 indicates that the voter intended to vote for Jacques Cartier for the officeof Mayor, and that mark 620 is, in fact, an overvote.The

10

15

20

25

30

35

40

45

50

55

60

65

US 9,202,113 B2 9

10

election official may update the vote recorded for the ballot 605 to include a vote for Jacques Cartier and include no vote for the officeofAlderman because the intention of the voter is not able to be discemed. Depending on the adjudication strategy used, an adjudication committee, for example, may have any number of terminals accessing the ballot optical images. With reference now to FIG. 7, an adjudication architecture 700 is described for a set of embodiments. An adjudication system 110-a may be coupled with network 115-a, and may receive optical image data for ballots identified as requiring adjudication. The optical image data may be provided by a central system, such as central server computer system 120 of FIG. 1, or may be provided directly from a ballot scanner such as optical scan ballot system 105 of FIG. 1 or 4. The adjudication system 110-a in this example includes a number of adjudication computers 705 through 715, that may be used to review ballot optical images, review audit marks associated with the ballots, and make determinations on whether particular marks on ballots should be recorded as votes. With reference now to FIG. 8, an adjudication architecture 800 according to another set of embodiments is described. In FIG. 8, an adjudication system 11O-b is coupled with network 115-b, and may receive optical image data for ballots identified as requiring adjudication. The optical image data may be provided by a central system, such as central server computer system 120 of FIG. 1, or may be provided directly from a ballot scanner such as optical scan ballot system 105 of FIG. 1 or 4. The adjudication system 11O-b in this example may be an example of one of the adjudication computers 705 through 715 of FIG. 7, may be a stand alone computer, or may be a computer that is connected directly to an optical scan ballot system. Adjudication system 110-b may be used to review ballot optical images, review audit marks associated with the ballots, and make determinations on whether particular marks on ballots should be recorded as votes. In some embodiments, adjudication system 110-b includes a receiver module 805, an adjudication module 810, a memory 815 that includes software 820, and a user interface 825. The receiver module 805 may include, for example, network interface hardware to allow connection with and communication over network 115-b. Such network interface hardware may include wired or wireless network interface cards and components, as are well nnderstood in the art. The adjudication module 810 may receive ballots to be adjudicated through the receiver module and may provide optical images of the ballots, and any included audit mark or other audit information, to the user interface 825. The user interface may include a monitor to display images to the election official, as well as a keyboard, mouse, or other data input device, as are well known. An election officialaccessing the user interface 825 may review the provided optical image and make a determination on votes for the ballot. The election official may provide input to the user interface 825, that the adjudication module 810 receives and uses to generate an adjudication mark. The adjudication module 810 may generate an optical image of the adjudication mark and append the adjudication mark to the ballot optical image, along with other relevant adjudication information. The optical image data including the adjudication mark may then be provided to the receiver module 805 and transmitted, for example, back to the central server computer system.Adjudication information may include adjudication of votes for a ballot, as well as information related to the date and time of adjudication, and identification of the election officialthat performed adjudication, for example.

Memory 815 may include random access memory (RAM) and read-only memory (ROM), and store computer-readable, computer-executable software code 820 containing instructions that are configured to, when executed (or when compiled and executed), cause the adjudication module 810 to perform various functions described herein (e.g., provide ballot optical images for review, receive adjudication information, append an optical image of the adjudication information to the ballot optical image, etc.). The components of the adjudication system 110-b may, individually or collectively, be implemented with one or more Application Specific Integrated Circuits (ASICs) adapted to perform some or all of the applicable functions in hardware. Each of the noted modules may be a means for performing one or more functions related to operation of the adjudication system 110-b. With reference now to FIG. 9, an example of a ballot optical image 900 following adjudication is described. In this example, the ballot optical image 900 includes an image of voter marked paper ballot 605-a and audit mark 610-a as illustrated in FIG. 6. As described above, the voter marked paper ballot 605-a includes two errors related to marks 615-a and 620-a. An election official may review the marks, and make an adjudication that mark 615-a is, in fact, a vote for Jacques Cartier, which is choice 3 on the ballot. The election official inputs this adjudication into the user interface of the adjudication system 110, and the adjudication system 110 generates image data 905 that includes the adjudication information. In the example of FIG. 9, the image data 905 of the adjudication information is appended to the ballot optical image data below the audit mark 610-a. In embodiments that do not include an audit mark image 610-a, image data 905 for the adjudication information may simply be appended to the ballot optical image data below the ballot optical image 605a, or at another suitable location adjacent the optical image of the ballot 605-a and/or audit mark 610-a. The adjudication information in image data 905, in this example, includes an identification of the user, namely the election official that performed the adjudication, an identification of the particular ballot question or office that was adjudicated, the change in the vote that was adjudicated, and a reason. In the example image data 905, it is noted that choice 3 was adjudicated (corresponding to Jacques Cartier) and that a marginal mark, corresponding to the ambiguous mark for that candidate, was added as a vote for the candidate. In such a manner, another user, election official, candidate representative, or observer may view the optical image and readily discern the image of the ballot as it was cast by the voter, the result of the initial optical scan of the ballot, and how the ballot was adjudicated. This can provide enhanced transparency and confidence in election results. With reference now to FIG. 10, an optical image 1000 of another ballot with audit and adjudication marks is described. In this example, a ballot optical image 1005, audit mark 1010, and adjudication mark 1015 are provided in optical image data. In this example, the optical scan ballot system that generated the audit mark 1010 identified two races as having over-votes. One resulting from mark 1020, and another resulting from a soiled area 1025 on the ballot 1005. Such soiled areas 1025 may be, for example, a coffee stain, food stain, or other type of soil on the ballot. Soiled ballots may be enconntered with increased frequency, for example, in mailin ballots. In this example, an election officialreadily discerns that mark 1020 is in fact an error on the ballot 1005, and that mark 1025, which resulted in the over-vote indication is a stray mark and not a vote. In this case, the election official adjudicated ballot 1005 to remove the stray mark and record the vote for that particular election area of the ballot 1005 to

10

15

20

25

30

35

40

45

50

55

60

65

US 9,202,113 B2 11

12

be for choice 3, corresponding to Jaques Cartier. It will be readily understood that the examples of FIGS. 9 and 10 are but two examples of common situations requiring adjudication. Other examples include votes for write-in candidates, overvoteswhere the voter also includes a note that a particular vote is not to be counted, marks that are next to a candidates name instead of in the marking area for the votes, etc. Additionally' multiple adjudications may be performed for any particular ballot, with each adjudication appended to the ballot optical image separately from other adjudications. In such a manner, an entire chain of adjudications may be viewed in the single optical image file, allowing a userto view the initial read of the ballot and each adjudication for the ballot. With reference now to FIG. 11, a system 1100 is described that includes central server computer system 120-a connected with network 115-c, according to a set of embodiments. The central server computer system 120-a may include a network interface 1105, an election information collection and management module 1110, memory 1115 that includes software 1120 stored therein, and a user interface 1125. The central server computer system 120-amay communicate with optical scan ballot system(s) and adjudication system(s), such as systems 105 and 110 of FIGS. 1, 4, and 7-8, through the network 115-c in order to receive and send information to such systems. The network interface 1105 may include, for example, commonly used network interface hardware to allow connection with and communication over network lIS-c. Such network interface hardware may include wired or wireless network interface cards and components, as are well understood in the art. The election information collection and management module 1110 may receive optical image data from optical scan ballot systems and store the data in memory 1115. Memory 1115 may include a database that is located locally and/or remotely from the central server computer system 120-a. Furthermore, central server computer system 120-a itself may include a single computer, or may include multiple computers which may be located remotely from one another. In any event, the election information collection and management module 1110 may receive optical image data and use this data to determine cumulative vote counts for one or more elections, and to perform election management tasks such as assigning ballots identified as having errors for adjudication, and assigning ballots to be audited according auditing procedures for the jurisdiction to verify the optical scan ballot systems appear to be properly tabulating votes. In one embodiment, the election information collection and management module 1110 receives optical image data for each scanned ballot, along with audit mark information for each ballot, and maintains appropriate vote counts based on the data. If the audit mark information for a ballot indicates one or more errors for the ballot, the election information collection and management module 1110 may assign the ballot to one or more election officialsfor adjudication. The election officials may adjudicate the ballots using an adjudication system 110 such as described with respect to FIGS. 1, and 7-8. The election information collection and management module 1110 may receive the adjudicated ballot information back from adjudication systems and update the vote counts for the various elections based on the adjudication information. The election information collection and management module 1110 may then store the optical image data, including the ballot optical image and the adjudication information, in memory 1115 for future retrieval as needed. The user interface 1125 may include a monitor to display images to the election management officials, as well as a keyboard, mouse, or other data input device as are well

known. An election official accessing the user interface 1125 may review optical images including one or more of audit mark information and adjudication information, as necessary. The election official may also use the user interface 1125 to monitor the status of ballots that have been assigned for adjudication and that have been received back at the election information collection and management module 1110 following adjudication. The user interface 1125 may also provide statistical information for use by election officials, as well as a number of other types of information for efficient election management. Memory 1115 may include random access memory (RAM) and read-only memory (ROM), and store computerreadable, computer-executable software code 1120 containing instructions that are configured to, when executed (or when compiled and executed), cause the election information collection and management module 1110 to perform various functions described herein (e.g., receive ballot optical images, identify images that require adjudication, receive adjudication information, maintain cumulative vote counts, etc.). The components of the central server computer system 120-a may, individually or collectively,be implemented with one or more Application Specific Integrated Circuits (ASICs) adapted to perform some or all of the applicable functions in hardware. Each of the noted modules may be a means for performing one or more functions related to operation of the central server computer system 120-a. In one embodiment, the election information collection and management module 1110 executes software that allows for management of the election process, including assigning ballots that require adjudication to one or more election officials. FIG. 12 illustrates a screen shot 1200 that may be provided to an election official operating an adjudication system, such as adjudication system 110 of FIGS. 1, and 7-8. Variousareas of the screenprovide information to the election official, such as an indication of ballots pending 1205, that indicates how many ballots are pending for adjudication by the election official.The election officialmay select a ballot to adjudicate, and is provided with an indication 1210 of contents of the ballot that are awaiting adjudication. Different races for the particular election are listed at 1215, with an area 1220 provided to indicate the particular race or question that requires adjudication. The election official may view an image of the ballot, an image of an audit mark for the ballot, and/or an image of a prior adjudication mark in area 1225. The election official may adjudicate the ballot by selecting a candidate that is determined to have been selected by the voter and marking a check box for the candidate, illustrated in this embodiment at 1220. Followingthe completion of adjudication the softwaremay update the information provided to the election official,illustrated for one example in the screen shot 1300 of FIG. 13. In this example, following the adjudication of the ballot of the example of FIG. 12, the indication of ballots pending 1305, is updated to indicate how many ballots remain pending for adjudication by the election official. The indication 1310 of contents of the ballot awaiting adjudication is also updated, to indicate, in this example, that no other areas of the ballot require adjudication. Different races for the particular election are listed at 1315, which may be expanded to view the recorded vote on the ballot at area 1320. The election official may view an image of the adjudication mark that they just generated, an image of the ballot, an image of an audit mark for the ballot, and/or an image of a prior adjudication mark in area 1325. In such a manner, ballots may be reviewed and

10

15

20

25

30

35

40

45

50

55

60

65

US 9,202,113 B2 13

14

adjudicated in a transparent and efficient manner, and that allows for efficientand effectivemanagement through central server computer system 120. With reference now to FIG. 14, a method 1400 for adjudicating ballots is described. The method 1400 may, for example, be performed by an adjudication system, a central server computer system, or optical scan ballot system of FIGS. 1, 4, 7-8, and 11, or using any combination of the devices described for these figures. Initially, at block 1405, optical image data comprising an optical image of a votermarked paper ballot is received. As discussed above, optical image data may be received from an optical scan ballot system that scans a voter-marked paper ballot, for example. One or more votes to be adjudicated on the voter-marked paper ballot are identified, according to block 1410. Suchvotes may be identified, for example, as votes that have ambiguous marks on the voter-marked paper ballot, elections or races in which no vote is entered (an 'undervote'), elections or races where too many candidates have been voted for (an 'overvote'), and/or the presence of a write-in candidate. Such votes also may be identified as a result of a damaged or misread ballot where no vote selections can be determined. In some embodiments, votes may be identified as a result of a ballot as a whole that is to be adjudicated. A ballot as a whole may be identified for adjudication in a number of situations, such as misread or damaged ballots as mentioned above, provisional ballots, absentee or mailed-in ballots that may have different formats, etc. Votes requiring adjudication may be identified, in some examples, by a central server computer system, with the optical image data and identified votes for adjudication provided to an adjudication system over a network. In some embodiments, votes requiring adjudication may be identified by an optical scan ballot system, with adjudication of the votes also performed with an adjudication system that is located with or near such an optical scanballot system. In still other embodiments, areas of multiple ballots may be identified to be adjudicated, such as when a significant number of ballots are identified as not having a vote for a particular race, for example, which may indicate that a system may not be properly reading the ballots. With continuing reference to FIG. 14, adjudication information for the one or more votes is received, as indicated at block 1415. Such adjudication information may include, for example, an identification of the election officialthat is adjudicating the ballot, a date and time of adjudication, information on how the one or more votes have been adjudicated, and/or reasons for the adjudication (e.g., ambiguous mark, stray mark, etc.). Adjudication information may also include, in some embodiments, identification of a computer being used for adjudication and a location of the computer.At block 1420, image data is generated that comprises the received adjudication information. Such image data may include all, or a subset of the adjudication information, and in some examples, non-image data may be included with the image data that includes all or a subset of the adjudication information. Finally, at block 1425, the image data comprising the received adjudication information for the ballot is appended to the optical image data. The optical image data then includes data that comprises an optical image of the voter-marked paper ballot and an optical image of the adjudication information that may be displayed together when the optical image data is displayed. In some embodiments, the optical image data is in a format that is widely known and used, such as, for example, .jpg, .tiff, .bmp, or .pdf format. Having optical image data in such a format allows the ballot optical image and the adjudication information optical image to be viewed on many different

platforms and using any of a number of widely available viewers, rather than requiring a proprietary viewer to view both the ballot and adjudication information. Storing the optical image data and adjudication information in the same optical data file also allows viewing of all of the pertinent election information related to the ballot without the need to access a separate database or data store that may have some of the information related to a ballot, such as adjudication information. With reference now to FIG. 15, another method 1500 for adjudicating ballots is described. The method 1500 may, for example, be performed by an adjudication system, a central server computer system, or optical scan ballot system of FIGS. 1, 4, 7-8, and 11, or using any combination of the devices described for these figures. Initially, at block 1505, optical image data comprising an optical image of a votermarked paper ballot is received. As discussed above, optical image data may be received from an optical scan ballot system that scans a voter-marked paper ballot, for example. One or more votes to be adjudicated on the voter-marked paper ballot are identified, according to block 1510. Such votes may be identified as described above, for example, as votes that have ambiguous marks, undervote situations, overvote situations, and/or the presence of a write-in candidate. Such votes also may be identified as a result of a damaged or misread ballot where no vote selections can be determined. In some embodiments, votes may be identified as a result of a ballot as a whole that is to be adjudicated, similarly as described above. Votes requiring adjudication may be identified, in some examples, by a central server computer system, or by an optical scan ballot system, with adjudication of the votes performed with an adjudication system that is located either locally or remotely. In other embodiments, areas of multiple ballots may be identified to be adjudicated, similarly as described above. With continuing reference to FIG. 15, an image of the ballot for review is provided to an election official, as indicated at block 1515. In some embodiments, adjudication is performed on an adjudication system that rnns specialized software the communicates with corresponding software on a central server computer system. The software at the central server computer system, in such embodiments, communicates ballots that require adjudication to the adjudication system, along with an identification of the votes on the ballot that require adjudication. At block 1520, adjudication information is received from the election official. The software at the adjudication system, in an example, provides the ballot and information related to votes requiring adjudication an election official who then adjudicates the ballot based on a visual review of the ballot. Adjudication information may include, for example, an identification of the election official that is adjudicating the ballot, a date and time of adjudication, information on how the one or more votes have been adjudicated, and/or reasons for the adjudication (e.g., ambiguous mark, stray mark, etc.). Adjudication information may also include, in some embodiments, identification of a computer being used for adjudication and a location of the computer.At block 1525, image data is generated that comprises the received adjudication information. Such image data may include all, or a subset of the adjudication information, and in some examples, non-image data may be included with the image data that includes all or a subset of the adjudication information. Finally, at block 1530, the image data comprising the received adjudication information for the ballot is appended to the optical image data. The optical image data then includes data that comprises an optical image of the voter-marked paper ballot and an optical image of the adju-

10

15

20

25

30

35

40

45

50

55

60

65

US 9,202,113 B2 15

16

dication information that may be displayed together when the optical image data is displayed. In some embodiments, as described above, the optical image data is in a format that is widely known and used, allowing the ballot optical image and the adjudication information optical image to be viewed on many different platforms and using any of a number of widely available viewers. Storing the optical image data and adjudication information in the same optical data file also allows viewing of all of the pertinent election information related to the ballot without the need to access a separate database or data store that may have some of the information related to a ballot, such as adjudication information. In some embodiments, one or more ballots may be adjudicated multiple times. For example, all ballots having write-in candidates may be adjudicated, with ballots having ambiguous marks, overvotes, undervotes, or other errors may be separately adjudicated. Thus, ballots having both a write-in candidate and one or more other errors may be adjudicated multiple times. In other examples, separate officials may adjudicate ballots independently, and in the event that the adjudications do not agree, ajudge may make a finalruling on how a vote is to be adjudicated. For example, representatives of each candidate on a ballot may provide separate opinions on how a vote is to be adjudicated, with non-matching opinions ruled on by an elections judge. With reference now to FIG. 16, a method 1600 for adjudicating ballots multiple times is described. The method 1600 may, for example, be performed by an adjudication system, a central server computer system, or optical scan ballot system of FIGS. 1, 4, 7-8, and 11, or using any combination of the devices described for these figures. Initially, at block 1605, optical image data comprising an optical image of a voter-marked paper ballot is received. As discussed above, optical image data may be received from an optical scan ballot system that scans a votermarked paper ballot, for example. At block 1610, vote information for the ballot is determined Such vote information may be determined as described above, for example, by performing a pixel-based analysis of voting and non-voting areas on the ballot to identify votes cast by the ballot, as well as any errors on the ballot, such as stray marks, overvotes, undervotes, and ambiguous marks. The vote information is appended to the optical image of the ballot in a humanreadable format, according to block 1615. At block 1620, one or more votes to be adjudicated on the voter-marked paper ballot are identified. Such votes may be identified as described above, as one or more votes that have an identified error, for example. An image of the ballot for review is then provided to an election official, as indicated at block 1625. In some embodiments, adjudication is performed on an adjudication system such as described above that runs specialized software as part of an election management software suite that runs across a central server computer system and one or more adjudication systems. At block 1630, adjudication information is received from the election official. Adjudication information may include, for example, an identification of the election officialthat is adjudicating the ballot, a date and time of adjudication, information on how the one or more votes have been adjudicated, and/or reasons for the adjudication (e.g., ambiguous mark, stray mark, etc.). Adjudication information may also include, in some embodiments, identification of a computer being used for adjudication and a location of the computer.At block 1635, image data is appended to the optical image data that comprises the received adjudication information. Such image data may include all, or a subset of the adjudication information.

Next, at block 1640, an image of the ballot for review is provided to a second election official. As mentioned above, the second election official may be a representative of a candidate, a representative of a political party, or a second election official focused on a particular type of adjudication such as adjudication of write-in votes. Adjudication may be performed on an adjudication system such as described above.At block 1645, adjudication information is received from the second election official. Adjudication information may include, for example, an identification of the second election official, along with one or more other items of adjudication information such as described above. At block 1650, image data is appended to the optical image data that comprises the received adjudication information from the second election official. Such image data may include all, or a subset of the adjudication information. In some embodiments more that two election officials may adjudicate a ballot, in which case the operations of blocks 1640 through 1650 are repeated for each additional election official.Thus, the optical image data includes an optical image of the ballot, and adjudication information from each election official that adjudicated the ballot. In some embodiments, as described above, the optical image data is in a format that is widely known and used, allowing the ballot optical image and the adjudication information from each election official to adjudicate the ballot to be viewed on many different platforms and using any of a number of widely available viewers. Storing the optical image data and adjudication information in the same optical data file also allows viewing of all of the pertinent election information related to the ballot without the need to access a separate database or data store that may have some of the information related to a ballot, such as adjudication information. It should be noted that the methods, systems and devices discussed above are intended merely to be examples. It must be stressedthat various embodiments may omit, substitute, or add various procedures or components as appropriate. For instance, it should be appreciated that, in alternative embodiments, the methods may be performed in an order different from that described, and that various steps may be added, omitted or combined. Also, features described with respect to certain embodiments may be combined in various other embodiments. Different aspects and elements of the embodiments may be combined in a similar manner. Also, it should be emphasized that technology evolvesand, thus, many of the elements are exemplary in nature and should not be interpreted to limit the scope of the invention. Specific details are given in the description to provide a thorough understanding of the embodiments. However,it will be understood by one of ordinary skill in the art that the embodiments may be practiced without these specificdetails. For example, well-known circuits, processes, algorithms, structures, and techniques have been shown without unnecessary detail in order to avoid obscuring the embodiments. Also, it is noted that the embodiments may be described as a process which is depicted as a flow diagram or block diagram. Although each may describe the operations as a sequential process, many of the operations can be performed in parallel or concurrently. In addition, the order of the operations may be rearranged. A process may have additional steps not included in the figure. Moreover, as disclosed herein, the term "memory" or "memory unit" may represent one or more devices for storing data, including read-only memory (ROM), random access memory (RAM), magnetic RAM, core memory, magnetic disk storage mediums, optical storage mediums, flash

10

15

20

25

30

35

40

45

50

55

60

65

US 9,202,113 B2 17

18

memory devices or other computer-readable mediums for storing information. The term "computer-readable medium" includes, but is not limited to, portable or fixed storage devices, optical storage devices, wireless channels, a sim card, other smart cards, and various other mediums capable of storing, containing or carrying instructions or data. Furthermore, embodiments may be implemented by hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof. When implemented in software, firmware, middleware or microcode, the program code or code segments to perform the necessary tasks may be stored in a computer-readable medium such as a storage medium. Processors may perform the necessary tasks. Having described several embodiments, it will be recognized by those of skill in the art that various modifications, alternative constructions, and equivalents may be used without departing from the spirit of the invention. For example, the above elements may merely be a component of a larger system, wherein other rules may take precedence over or otherwise modify the application of the invention. Also, a number of steps may be undertaken before, during, or after the above elements are considered. Accordingly, the above description should not be taken as limiting the scope of the invention. The invention claimed is: 1. A method comprising: identifying one or more votes to be adjudicated on a votermarked ballot; providing an image of the voter-marked ballot for review by one or more election officials; receiving adjudication information from the one or more election officials, wherein the adjudication information includes a determination whether a particular mark on the ballot should be recorded as a vote; generating image data comprising the received adjudication information; and appending the image data comprising the received adjudication information for the ballot to the optical image data. 2. The method of claim 1, wherein the adjudication information comprises: an identificationof an election officialperforming the adjudication; and a record that one or more votes on the ballot have been adjudicated. 3. The method of claim 1, wherein the adjudication information further comprises a date and time of adjudication. 4. The method of claim 1, wherein appending the image data comprising the received adjudication information for the ballot to the optical image data comprises: inserting text in a margin of the optical image of the ballot, the text including the adjudication information. 5. The method of claim 1, further comprising: displaying the optical image data including an image of the ballot and adjudication information appended to the image of the ballot. 6. The method of claim 1, wherein identifying one or more votes recorded on the voter-marked paper ballot to be adjudicated comprises: evaluating a target area on the ballot; and determining that a mark within the target area is ambiguous. 7. The method of claim 1, wherein identifying one or more votes recorded on the voter-marked paper ballot to be adjudicated comprises: determining that the ballot as a whole is to be adjudicated.

8. The method of claim 1, wherein identifying one or more votes recorded on the voter-marked paper ballot to be adjudicated comprises: determining that one or more portions of the ballot include a different number of votes than expected. 9. The method of claim 1, wherein identifying one or more votes recorded on the voter-marked paper ballot to be adjudicated comprises: determining that one or more portions of multiple ballots include one or more votes to be adjudicated. 10. The method of claim 1, wherein identifying one or more votes recorded on the voter-marked paper ballot to be adjudicated comprises: inspecting the ballot image to determine if a write-in vote has been selected; and assigning the write-in selection to a candidate. 11. The method of claim 1, wherein identifying one or more votes recorded on the voter-marked paper ballot to be adjudicated comprises: receiving an indication that one or more votes recorded on the voter-marked paper ballot require adjudication. 12. A system for adjudication of voter-marked paper ballots, comprising: a receiver module configured to receive ballot optical image data comprising an optical image of a votermarked paper ballot; and an adjudication module configured to receive adjudication information from one or more election officials, wherein the adjudication information includes a determination whether a particular mark on the ballot should be recorded as a vote, generate image data comprising the received adjudication information for the ballot and append the image data for the ballot to the optical image data. 13. The system of claim 12, wherein the adjudication module is configured to receive an indication that the ballot requires adjudication when a mark within a target area of the ballot is determined to be ambiguous. 14. The system of claim 12, wherein the adjudication module is configured to receive an indication that the ballot as a whole requires adjudication. 15. The system of claim 12, wherein the adjudication information comprises one or more of: an identificationof an election officialperforming the adjudication; a record that one or more votes on the ballot have been adjudicated; or a date and time of adjudication. 16. The system of claim 12, wherein the image data comprising the received adjudication information for the ballot is inserted as text in a margin of the optical image of the ballot. 17. A system for adjudication of voter-marked paper ballots, comprising: means for identifying one or more votes recorded on a voter-marked paper ballot to be adjudicated; means for receiving adjudication information from the one or more election officials, wherein the adjudication information includes a determination whether a particular mark on the ballot should be recorded as a vote; means for generating image data comprising the received adjudication information; and means for appending the image data comprising the received adjudication information for the ballot to the optical image data.

10

15

20

25

30

35

40

45

50

55

60

65

US 9,202,113 B2 19 18. The system of claim 17, further comprising: means for providing an image of the ballot for review by an election official; and means for receiving adjudication information from the election official. 19. The system of claim 17, wherein the means for incorporating the adjudication information into the ballot optical image file comprises: means for inserting text in a margin of the optical image of the ballot, the text including the adjudication information.

* * * * *

20

10

111111

(12)

United States Patent

1111111111111111111111111111111111111111111111111111111111111 US009710988B2

Patent No.: US 9,710,988 B2 (45) Date of Patent: Jul. 18, 2017 (10)

Poulos et al. (54) MARGINAL MARKS WITH PIXEL COUNT (75) Inventors: John Poulos, Toronto (CA); James Hoover, Montreal (CA); Nick Ikonomakis, Toronto (CA); Goran Obradovlc, Toronto (CA) (73) Assignee: Dominion Voting Systems Inc., Denver, CO (US) ( *)

Notice:

USPC 705/12 See application file for complete search history. (56)

U.S. PATENTDOCUMENTS 6,250,548 Bl Al Al Al

200210091673 200210143610 2003/0052981

Subjectto any disclaimer,the term of this patent is extended or adjusted under 35 U.S.C. 154(b)by 726 days.

348/272

Requisition by the Examiner for Canadian Application No. 2,466,466 dated Aug. 3, 2015, from the Canadian Intellectual Property Office.

Prior Publication Data US 2012/0259680Al

(Continued)

Oct. 11, 2012

Primary Examiner - Chrystina Zelaskiewicz (74) Attorney, Agent, or Firm - Holland & Hart LLP

Related U.S. Application Data (63) Continuationof applicationNo. 13/463,536,filed on May 3, 2012, which is a continuationof application No. 11/121,997,filed on May 5, 2005, now Pat. No. 8,195,505. (30)

Foreign Application Priority Data

May 5, 2004

(CA)

705/12

G06T 3/4007

OTHER PUBLICATIONS

Jun. 15, 2012

(65)

* *

6/2001 McClure et al. 7/2002 Seibel et al. 1012002 Munyer 3/2003 Kakarala

(Continued)

(21) Appl. No.: 13/525,187 (22) Filed:

References Cited

2466466

(51) Int. Cl. G07C 13/00 (2006.01) G06Q 20/38 (2012.01) G06Q 20/40 (2012.01) (52) U.S. Cl. CPC . G07C 13/00 (2013.01); G06Q 20/382 (2013.01); G06Q 20/401 (2013.01) (58) Field of Classification Search CPC G07C 13/00

(57)

ABSTRACT

A system, method and computer program for tabulating votes and creating an audit trail is provided. A ballot processing device may include a paper feed mechanism, a computer, a ballot processing application loaded on the computer,and a digital scanning device linked to the computer. The ballot processing application may process the digital image to establish a series of processing results definingone or more voting results for the paper ballot, and also an audit trail. The ballot processing application may process the digital image to define the voting results based on criteria established by election officials, including ambiguous mark criteria. The audit trail enables election officials to verify that particular paper ballots have been processed correctly in accordancewith these criteria. 20 Claims, 7 Drawing Sheets

US 9,710,988 B2 Page 2 (56)

References

Cited

U.S. PATENT DOCUMENTS 2003/0173404 Al 2007/0170253 Al

*

912003 Chung et al. 7/2007 Chung

G06K 7/10346

235/386

OTHER PUBLICATIONS Election Systems and Software, Inc., ES&S Model 100 Precinct Ballot Counter Operator's Manual, 101 pages, Apr. 2000.t

* cited by examiner t cited by third party

u.s.

Patent

Jul. 18, 2017

Sheet 1 of 7

US 9,710,988 B2

Voter Enters Voting Location

Voter deemed ineligible

t

Voter cannot vote Does the voter require an audio or paper ballot?

Print the required ballot 'on-demand' on the local printer and issue to voter

Election official to place marked ballot is put into the 'Spoiled Ballots' envelope, Voter is issued a new ballot and is reo instructed on ballot marking

Tabulator to ask user. Are the overvotedl blank races intentional?

Yes

No

L.-

Yes

Does the tabulator detect any ambiguous markings, or is the DRO box blank?

Does the tabulator detect any overvotes or blank or blank races?

I

ToFi9,1b

Audio

Iss~e prepnnted paper ballot to voter

1 Ballot feeds completely through the tabulator and falls into the ballot box Tabulator is ready for next .paper ballot

FIG.1a

u.s.

Patent

From Fig. 1a

Jul. 18, 2017

US 9,710,988 B2

Sheet 2 of 7

Voter is escorted to the audio voting area where he is given a set of earphones and a handheld pendant thumb switch Once the voter is set up, the election official selects the proper audio ballot on the tabulator to initialize the voting session The voting session begins from the beginning and the voter 'navigates' through the ballot selecting as many candidates as desired The ballot is cancelled and the voter returns to the start of the ballot to begin again

At the end of the audio voting process, the voter is prompted to review the selected choices one last time and to confirm the selections

Were the selections made confirmed by the voter?

>---

NO----j

Yes The tabulator makes an audible sound to alert the election official thai the audio vote has been completed

No The election official assist the voter in leaving the audio voting area

.r<:

The election official asks the voter: Are you happy with the ballot- do you indeed wish to cast the ballot? The election official prompts the tabulator to accept the ballot, and a paper reciept is printed and falls into the ballot box

FIG. 1b

Tabulator is ready for the next ballot

u.s.

Patent

Jul. 18, 2017

US 9,710,988 B2

Sheet 3 of 7

10

o Ta~:~~r } "__3~~ ,~,~""""""_ 22

24

14

~

<;

Path

--- --- ----- ----,

~

:,

~

Auxiliary Ballot Box

,, , ,,

Tabulator Ballot Box

IUUr FIG.2

12

~16

u.s.

Patent

US 9,710,988 B2

Sheet 4 of 7

Jul. 18, 2017

/

m

~

m m m m m m m m m\~

!.Z":2l

General Municipal Election

!.Z":2l

January

!.Z":2l

!.Z":2l

BALLOT FOR THE OFFICE

E22Zl

18 'I::: ::--

OF

f2Ll

BALLOTFOR THE OFFICE OF

MAYOR

E22Zl

Vote for ONE(1)1st dloice

!.Z":2l

1)RobertL

& ONE (1)2nd choice, as shown at right

~

-2l.Jean !'ZZZ:l

3)Jacques

E22Zl

4} Samuelde

!'ZZZ:l

51 James

E22Zl

6)Leit

!'ZZZ:l

7)John

E22Zl

8) Henry

!'ZZZ:l

9) Jake

!'ZZZ:l

10)William l.M.

!.Z":2l

11)Wilfrid

!'ZZZ:l

12)John A.

!.Z":2l !'ZZZ:l

13)LesterB. 14)LouisS.

:: I 1"1 m

COOK

3) Clause

ERIKSSON

0

$

CHAMPLAIN

FRANKLIN

0

HUDSON

0

JOLIET

0

KING

0

LAURIER

0 0

MacDONALD PEARSON SAINT·LAURENT

_n

f2Ll f2Ll f2Ll

o.

Candidate 1 00 Candidate 2 • 0 Candidate 3

f2Ll

ea

,~ ",
0 0 0

0

0

n

-'01

TRIIDEAU

P<mE

•

CABOT CARTIER

f2Ll

COUNCILLOR

Vote for only ONE (1)candidate

BORDON

f2Ll

11891

onext to your choicets), like this:

To vote, completely fill in the box(es)

E22Zl

f2Ll

,

ORO /

City of Anywhere

!.Z":2l

m

1-

1)MilesJ.

O'DANIEL VON ESPIE

..!:!l .£!: .::; ..!::i! r- :z <s: ." s::: Q)

'"=> -<

:::> '<

<>

'<

;;;::

g,

iii'

~


*"Q

m

~. ""'=> -<" '" '"

f2Ll

00 00 00

MASSEY

2) MenelausP.

...,

f2Ll @

I2Zl @

...... co

f2Ll

-n ~~

s:::

"0

o· ..,,0

- ...... °0 ..,,-1

f2Ll

m;:a

6'

I2Z.l

-I ::I: :;u

m m

;:a 0

en m :z -rt m r0

;::0::

-n 0 ::0 0

:z -I G)

0

s: m ::0 -<

@

n

=> '"

0

n

Q.

s: <> (")

rr: c::

:z

C>

a: !!l.

'"

s: ("") » »

z (") m

V>

FIG. 3

~

@

I:}

f2Ll

::0

rnJ

•

s » Z

:;:0

~ ~

» m

::0

nn 0 n n nl

m m m

f2Ll

~

s: -0

20

I

rnJ rnJ rnJ

m

~

~

\

17

u.s.

Patent

m

~

US 9,710,988 B2

Sheet 5 of 7

Jul. 18, 2017

m

~ ~ ~

m

~ ~ ~

I22Zl To vote, completely fill in the box(es)

I22Zl

I22Zl

BALLOTFOR THE OFFICE OF

I22Zl

Vote for only ONE (1)candidate

"t--122Zl ~

1) Robert L.

Candidate1 DO Candidate2 • 0 Candidate3 O.

CARTIER

0 0 0

4) Samuelde 5}James

I22Zl

6) lei!

ERIKSSON

0

I22Zl

7) John

FRANKLIN

0 0 0 0 0 0

8) Henry

CHAMPLAIN COOK

HUDSON JOLIET

9) Jake

t:221

10)William l.M.

I22Zl

11)Wiffrid 12)John A.

13) Lester B.

KING LAURIER MacDONALO n~ A n~I'\"1 rt;;;I-\"oJvn

I22Zl

14) Louis S.

SAINT-LAURENT

I22Zl

15)PierreE.

TRUDEAU

I22Zl

I

Nick Salvator

"'~?:.

,<::'-

I22Zl

I22Zl

COUNCILLOR

& ONE (1)2nd choice, as shown at right

I22Zl

I'///l

BALLOT FOR THE OFFICE OF Vote for ONE (1)1S1choice

BORDON

•

next to your choice(s), like this:

CABOT

3) Jacques

I22Zl

o

MAYOR

~Jean

I22Zl

I22Zl

jj·s.1

General Municipal Election January 1 1891

I22Zl

18

~

ORO

City of Anywhere

I22Zl

I22Zl

~

IZ2J

I22Zl

.-. LJ

0 0

1) Miles J.

MASSEY O'DANIEL

2) MenelausP.

VON ESPIE

3) Clause

.e -rr

-!:!!

=> :;:>

-<

",

""

;s::

ru

.;e r-'

c:

n

'<

~ ~ z l:ll:: zs:

?l'!

<-



:::I '" :::I '"

II 0 0 0 0

•

-


m

~ ~~F!

~.

Sf

...;;:

C -0

0'

o·

~~ ~~

IZ2J IZ2J IZ2J IZ2J IZ2J IZ2J IZ2J E223 E223 @

IZ2J IZ2J IZ2J IZ2J

m m

IZ2J

2 '" a:

@

::0

:g :;;:: 0

;;0

0

(I)

m Z

.."

m

.-<::>

"

(') ::>:

..,.. 0

:::0

z

Q.

G)

:;;:: ...,

0

("')

-i

:;;:: m ;;0 -<

'" ~

:;;::

~ ~ ~ Z ("')

G)

'" IJiIllIilOIWO

m

'"

;;0 ;;0

» .-

@

0

@

:::0

@

m

:s: » :z

IIJi

@ @

IZ2J

~

~ ~ ~ ~ ~ ~ ~ ~

ORO: 2109 Office of Mayor: 0,0,0,1,0,0,0,0,1,0,0,0,0,1,0,780 Vote for: Nick Salvator . Office of Councillor: (800,0), (1,790), (0,0) Vote for: Miles J. MASSEY (1st), Menelaus P. O'OANIEl Office of ALDERMAN: Vote for: OVER-VOTE

IZ2J

:x:

-i

I22Zl

~

IZ2J

(2nd)

2,812,2,823,788,774

1

FIG. 4

( 30

~

'\

u.s.

Patent

Jul. 18, 2017

US 9,710,988 B2

Sheet 6 of 7

Data Card from Data Card from Data Card from Tabulator 1 Tabulator 3 Tabulator X Data Card from Data Card from te::]o.. l.rnj_j Tabulator 2 Tabulator 4

rt.::=.n

rt.::=.n

UOJ

~

UOJ

~

Computer Central Election Database

32

l

( 28

Q Q

Q

Auditing PC Auditing PC Station #1 Station #2

Auditing PC Auditing PC Station #3 Station #Y

FIG. 5

I

u.s.

Patent

~ ~

~

To vote, completely fill in the box(es) BAlLOT FOR THE OFFICE OF

MAYOR

VoteforonlyONE(1)candidate 1) Robert L.

~

2) Jean

~

3) Jacques

~ ~

4) Samuel de

~

6) Leif

~ ~

0 0 0 0 0 0

BORDON CABOT CARTIER CHAMPLAIN

5) James

COOK ERIKSSON

7) John

~

~

~

~

~

~

ORO

BALLOTFOR THE OFFICE

9) Jake

JOLIET

10) William L.M.

~

11)Wilfrid 12) JohnA. 13) Lester B.

~

14) louis S.

~

15) Pierre E.

~ ~ ~

!LLLLLLJ

NiCK

KING LAURIER MacDONALD PEARSO~J SAINT·lAURENT

~

1) Miles J.

TRUDEAU

Salvator

.9

~,- ~ .!:::? ~ ::z <- m s: c:: §!. n ~ ;;;:

< 0"'" OJ o "'Tl:Z:: 'l> ;;m,0' Q c:

Co>

::l

n

(")

;>;;

00

VON ESPIE

~

z

m

o.

O'DANIEL

3) Clause

o»

,-

'),~?:,

1110

MASSEY

2) Menelaus P.

0 0 0 0 0 0 0

Candidate1 DO Candidate2 • 0 Candidate3 O. ......

'<

L....J

COUNCILLOR

VoteforONE(1) 151choice & ONE(1)2ndchoice,as shown at right

0 '"=>

HUDSON

•

fZZ)

OF

:::I

8) Henry

1;·5. I

fZZ)

o next to your choice(s), like this:

-r-r

FRANKLIN

~ ~ ~

~

January 1 1891

~ ~

~

~

GeneralMunicipal Election

~

~

~

fZZ)

~

~

~

City of Anywhere

~

~

US 9,710,988 B2

Sheet 7 of 7

Jul. 18, 2017

::::;9

..,

'"

S --I :r ::u

00

0

m

" ......

m

a

0

0

,-

;:0

--I

Cl 0 :;:

m ;;0 -<

» r

es; n; '"

s: n c: » z » ::z ;:0

U>

C)

CJ m

r-

Cl

n m

fZZ) fZZ)

fZZl fZZl fZZ)

fZZl fZZl fZZ) fZZ)

fZZl

'"

:::l C.

s: .,..

fZZ)

fZZ)

<>

s:

z

fZZ)

fZZ)

~

0

fZZ)

m;;u

m m

;;0

fZZ)

;0

;:0

3:

» z

1II~IIiO~D

fZZ) fZZ) fZZ)

fZZl fZZ)

fZZl

m

~

~

~

~

~

~

~

~

ORO: 2109 Office of Mayor: 0,0,0,1,0,0,0,0,1,0,0,0,0,1,0,780 Vote for: Pierre E. TRUDeau Office of Councillor: (800,0), (1,790), (0,0) Vote for: Miles J. MASSEY (1 st), Menelaus P. O'DANIEL

(2nd)

Office of ALDERMAN: 2,812,2,823,788,726 Vote for: OVER-VOTE

FIG. 6

30

"'\

l__ 17

US 9,710,988 B2 1

2

MARGINAL MARKS WITH PIXEL COUNT

One of the challenges in permitting efficientverification of voting results, and also demonstrationof the integrity of the vote, are the subjectiveelements generally involved in determiningvoter intent. In accordance with numerous prior art voting technologies, the assessmentof what constitutes a vote in favour of a particularcandidateor a particular answerto a questionis determinedby subjectiveassessment. This is best illustrated in the context of a manual hand count. Historically, voters would select their candidates using pen and paper, making a mark (an 'X' for example) beside the candidate(s) of their choice. This paper ballot would then be manually counted by election officialswith candidatesand their agents allowedto scrutinizethe operation. However, by the very nature of elections, different parties involvedwith electionshave very differentinterests. A set of candidates and their agents rarely share common motivation and accordingly the interpretation process of what constitutes a mark on a ballot has always been prone to inconsistencies.This fact generally makes it difficultto achieve the same tally when counting the same ballots multipletimes, as there is no assurancethat any givenballot wouldbe interpretedthe sameway by everyinterestedparty. The reason for this is straightforward:often the voter would make a mark that couldbe deemedas ambiguousby anyone of the people involved with the manual ballot count (i.e. electionofficialor candidateagent).Whatmay be obviousto one agent may not be obviousto another agent. In addition, many ballots were deemed 'spoiled' or 'blank' because the voter (eitherintentionallyor unintentionally)made too many or too few marks on the ballot. The apparent lack of objectivity was addressed in a numberof prior art systems.The MechanicalBased Systems and Optical-MarkBased Systemseither incrementeda particular vote or not based on objective standards. The Mechanical Based Systems have been deemed susceptible to inconsistencies in well publicized cases. For example,somemechanicalpunch systemshave been shown to leave cut-out 'chads' partially attached, leading to disputes surroundingelection results. Also, in the case of Mechanical Based Systems and Optical-Mark Based Systems, the objective standards for what constitutes an indication of voter intention are generally set by the manufacturerof the voting device and have not been adaptableto conformto objective standardsdetermined by election officials for a particular vote. For example, in Optical-Mark Based Systems an integration thresholdis usually definedto establishhow dark a mark has to be in order to constitutea mark recognizedby the system as being such. This integrationthreshold is permanentlyset by the manufacturer. These concerns have hindered the ability to verify vote results,and have generallyhad a negative impacton impressions of vote integrity. To some extent these problems have been addressed by DRE systems.Voter intention in a DRE system is a binary input from a digital device so ambiguity cannot exist. However,it is well known that processingerrors, computer memory corruption or computer memory loss can occur in digital devices and there have been well publicized reports suggestingthat this may be the casewith someprior art DRE systems. If this type of failurewere to occur,voter intention is permanently corrupted or lost and a new election would have to be held. In addition to the aforesaid disadvantages,the prior art solutions generally do not enable an election official to effectivelyaudit voting results and/or verify the integrity of

This applicationis a continuationof U.S. patent application Ser.No. 13/463,536 which was filed on May 30, 2012, and entitled"System and ComputerProgramfor VoteTabulation with an ElectronicAudit Trail"; which is a continuation of U.S. patent application Ser. No. 11/21,997, filed May 5, 2005, now U.S. Pat. No. 8,195,505,issued Jun. 5, 2012, and entitled"System,Methodand ComputerProgram for VoteTabulationwith an ElectronicAudit Trail"; which claims priority to Canadian Patent Application No. 2,466, 466, filed May 5, 2004, and entitled "System, Method and Computer Program for Vote Tabulationwith an Electronic AuditTrail";the entiredisclosuresof which are incorporated herein by reference.

10

15

FIELD OF INVENTION This inventionrelates generallyto technologiesfor tabulating votes. This invention relates more particularly to ballot boxes with vote tabulationcapabilities.The invention also relates to method for tabulating votes in an election. BACKGROUNDOF THE INVENTION Numerous voting technologies are known. These technologies are generally directed to permitting votes to be recorded with efficacywhile maintainingthe secrecy of the ballot. One categoryof votingtechnologyinvolvesthe use of the paper ballot. Variousmethods exist for collecting and tabulating votes recorded by paper ballots. Prior art has been created to rely on paper ballots while using modern digital technology. For example, "Optical-mark Based Systems" allow voters to mark paper ballots and feed the ballots through an automated optical ballot scarmingtabulator into sealed ballot boxes. The optical scanningtechnology determines the nature of the vote and the computer technology provides a means to quicklytabulate results. The results are transmitted to a central computer by a variety of transmission technologies.All results are tallied at the central computer. Another category of voting technology is best described as "MechanicalVoting Systems". These systems generally utilize mechanical counters for counting votes. These systems include lever machines and punch card ballots. Another category of voting technology is the "Direct Recording Electronic" (DRE) system. This type of system permits voters to enter their vote into a digital system by means of an electronic interface such as a touch screen, mouse or scrolling cursor. Generally speaking, an interface device in each voting stationis linkedto a computersystem. The results compiledby each set of voting stations are then transmittedto a central computer system using a variety of computer architecturesand transmissionmodes. In a variation of the DRE system,the entry of a vote into a digital system is linked to a paper receipt that is used as a voter verifiedpaper receipt. This paper receipt can be used in a paper-based audit procedure-an operation that is impossible in a true DRE system. There is a need for voting technologiesthat provide the ability to audit the election results. Democracydepends on accurate and incorruptibletabulation of voter intent; so by extension,voting technologiesmust be able to demonstrate vote tabulation integrity.

20

25

30

35

40

45

50

55

60

65

US 9,710,988 B2 3

4

the vote. For example, the above mentioned prior art technologies do not permit an electionofficialto pick a ballot at random and examine the results of the optical scan algorithms that determinedvoter intent. This ability to compare the paper ballot with the results collected by a vote system allows the election official to verify the accuracy of each votingunit and ensurethat there are no randomor systematic tabulation errors in the voting system or subsystems. There is a need for a voting system and ballot collection apparatusthat enablesobjectivemeasurementof voter intention and a method to verify accurate data collection to demonstratethe integrity of the vote.

cartridges from a plurality of ballot boxes is retrieved and stored to a central memory linked to the central processing computer. One aspect of the present inventionis that the association of the vote stamp with the corresponding digital image enables rapid and cost effectiveverificationof the integrity of the processes. For example, a random selection of a definedpercentage of ballots processed in accordancewith the invention can be examined for accurate collection of voter intent by calling up from the central memory the digital images and correspondingvote stamps. (for example from ballots received in a particularpolling station) and by comparingthe digital image displayedon a computerscreen with the results of the processing thereof as shown by the data included in the vote stamp. Another aspect of the present inventionis that it provides the objectivity and speed of a digital process in connection with the use of paper ballots in an electoral process. A writtenrecordof the electoralchoicein the form of the paper ballot is maintained as well as a cost savings as fully electronicvoting (for example using touch screens) is generally more costly. One object of the present inventionis to provide a system and method that enables an electoral process that can be audited efficiently,accurately and transparently. Another object of the present invention is to include the aforesaid auditing capabilities in a machine that can be easily accessed by any persons with visual, motor or language disabilities.

10

SUMMARYOF INVENTION 15

A tamper-proofand secure ballot box includes a slot that permits paper ballots (which include one or more voter selection entry areas to be manually completed by a voter using a suitable marking device) to be fed into the box by means of a paper feed mechanism. 20 The interior of the box includes a ballot processing device. The ballot processing device includes a computer and a digital scarmingdevice. The ballot processing application of the present invention is loaded on the computer. The digital scanningdevice scans each paper ballot fed into 25 the box to create a digital image thereof, in a manner that is known. The ballot processing application analyses each digital image based on predeterminedcriteria. The predetermined analysis criteria include co-ordinatesof marking locations, 30 co-ordinates of ballot identifiers, locations of relevant scripts, and pixel-based threshold values for determining voter intent (marked, unmarked, ambiguous). The ballot box includes an alarm device that signals incorrect ballots or when a ballot has been determined to 35 require resolution of intent by the voter (erroneous or ambiguouslymarked voter selection entires). In this case, the ballot is returned by the feeding mechanism, following which normal procedures are followed to cancel the ballot and issue a new ballot to the voter. 40 Another aspect of the inventionis that the ballot processing applicationcreates a "vote stamp" based on analysis of each digital image. This vote stamp is attached to a digital image of the scanned ballot and stored as a single data file to a memory linked to the computer.The vote stamp may 45 includea randomlyassignedunique identifiercorresponding to a particular digital image of a ballot. The vote stamp also may includedata correspondingto the results of the analysis of the digital image conducted by the ballot processing application.Another aspect of the vote stamp is that it may 50 includeother datarelevantto the electionprocess such as the polling station where the particular ballot was scanned. The paper ballot is retained within the ballot box of the present invention in a secure marmer and then handled in accordancewith the specificrules of the election in which 55 the ballot box is used. The various data files correspondingto the digital ballot images and corresponding vote stamp are stored in the memory in random order.The voting selectionof a particular individualcannot be discernedby individualsobserving 60 the order in which ballots are fed into the ballot box. The computer also includes a suitable back-up storage device and related utilities to reduce the risk of data loss from the memory. The systemof the inventionincludes a centralprocessing 65 computer.The memory linked to the computerconsistsof a computer cartridge. Data from a plurality of computer

BRIEF DESCRIPTIONOF THE DRAWINGS A detailed description of several embodiments of the present invention is provided herein below by way of example only and with referenceto the followingdrawings, in with: FIG. 1 is a flowchartillustratingthe methodof the present invention,in accordancewith one particular aspect thereof. FIG. 2 is an illustration of the tabulation unit of the present invention, in a cross-sectionthereof. FIG. 3 is an illustrationof a paper ballot that provides an example of a paper ballot processed in accordancewith the present invention. FIG. 4 illustrates a scanned and processed image of the paperballot illustratedin FIG. 3 that includesthe vote stamp footer of the present invention. FIG. 5 is a diagram illustrating a system architecture representative of one embodiment of the system of the present invention. FIG. 6 is a further scannedimage of anotherpaper ballot that includes a vote stamp footer, wherein the vote stamp footer reflectsprocessingresults inconsistentwith the voter intention. In the drawings,preferred embodimentsof the invention are illustrated by way of example. It is to be expressly understoodthat the descriptionand drawingsare only for the purpose of illustration and as an aid to understanding,and are not intendedas a definitionof the limits of the invention. DETAILEDDESCRIPTIONOF THE INVENTION One aspect of the present invention IS a method that includes the following steps: 1. Establishing one or more criteria for analyzing a plurality of paper ballots, the paper ballots including one or more voter selection areas, which criteria

US 9,710,988 B2 5 include threshold values for determining whether a particularvoter selectionareahas been filledin, has not been filled in, or it is ambiguouswhether or not it has been filled in; 2. Scanning at least one of the paper ballots using a scanning device, the scanning device generating a digital image document of the each paper ballot; 3. Analyzing the digital image document to determine whether a particularvoter selectionarea has been filled in, has not been filledin, or is ambiguousas to whether 10 or not it has been filled in, so as to define a series of processingresults for each digital image; 4. Creationof a vote stampwhich includesthe processing results and attachingthe vote stampto the digitalimage 15 document; and 5. Comparingthe processingresultswith the digitalimage document corresponding to a paper ballot, thereby enabling the election results to be audited on a per ballot basis. A further aspect of the method of the present inventionis 20 that the threshold values are based on pixel count. The threshold values are established by election officials, as particularizedbelow. In anotheraspectof the invention,the vote stampincludes other data relevantto the electionprocess such as the polling 25 station where the particular ballot was scanned. Furtheraspectsof the method of the present inventionare explainedbelow. Overall, the method of the present invention is illustrated in FIG. 1. FIG. 2 illustratesone particularembodimentof the inven- 30 tion which consists of a hybrid paper/electronicvote tabulator 10. The hybrid paper/electronicvote tabulator or tabulation unit 10 includes a lock down type ballot box 12, earphones 14, and a pendant manual trigger 16. As voters come into the polling location, they are gener- 35 ally processedby the election officialwho determinesvoter eligibility (based on local election rules), and also determines the proper ballot for the voter. Based on the voter's requirements;they are either issued a paper ballot, or issued an 'audio' ballot, in accordance with one aspect of the 40 present invention. FIG. 3 illustrates a representativepaper ballot 17, which includes a plurality of voter selection areas 18. In the case of the particular paper ballot 17 illustrated,there is also an election officialarea 20, which appears in most ballots, and 45 is used by the election official to validate the particular ballot. Usually the validation takes the form of a signature in the election official area 20 by the election official, however, other means of authentication can also be used. The election officialarea 20 servesto mark the paper ballot 50 17 as having been officiallyissued. It shouldbe nnderstoodthat the paperballots 17 can either consist of pre-printed ballots or a ballots printed locally on-demandby means of a regular laser printer. The voter will generally take the validated blank paper 55 ballot 17 to a private area (such as a desk with privacy barriers), and make a mark or marks (this can be done in numerousways such as fillingin a box, making an 'X', etc) beside the chosen candidate(s)with a permanent marker. For the sake of clarity,this disclosuregenerallyrefers to 60 selectionof a candidateor candidates,however,some votes relate to other questions such as referendum questions and the like. The present inventionextendsto votes obtainedfor a voter mandateon all matterswhethercandidates,weighted candidate votes, referendum questions and the like. 65 The voter then generallytakes the markedpaper ballot 17 to the tabulationunit 10, where it is fed through the hybrid

6 tabulator 10, as shown in FIG. 2. Secrecyof voter intent is preferablymaintainedby using a confidentialityfoldermade from paper (not shown). The tabulation nnit 10 includes a ballot processingdevice22. The ballot processingdevice 22 includes a computer (not shown) and a ballot reader or digital scanningdevice 24. The ballot processingdevice 22 also includesa known paper feed mechanism26. The ballot 17 is fed through the paper feed mechanism 26 and the digital scanning device 24 takes a high resolution optical scan of the ballot 17. In one particular aspect of the invention, the tabulation unit 10 takes the resulting high resolutionscannedimage (not shown)of the entireballot 17, and saves one copy of this image to non-volatile memory (not shown) linked to the computer (not shown) so as to create a permanent record thereof. Loaded on the computeris the ballot processingapplication 28 which constitutes the computer program aspect of the present invention.The ballot processing application 28 randomly assigns a filename to each scanned image (not shown)to ensurethat the order in which the ballots 17 were scanned remains private. The ballot processing application 28 also sendsanother copy of the images (not shown)to the computer (not shown) for image recognition. In a particular aspect of the invention,the computer (not shown) is linked to the digital scanning device 24, and the ballot processing application 26 defines an image recognition routine (not shown) that is applied to the digital scanning device 24. One aspect of the image recognitionroutine (not shown) is that the ballot processing application 28 enables the computerto selectivelyrecognize specificareas of each image (not shown) and analyze such specificareas as particularized below to define a series of processing results (not shown) associatedwith the particularballot 17. For example,the ballot processingapplication28 enables the computerto analyzethe various securitymarkingson the ballot 17 to ensure it is a valid ballot, perform a pixel count checkto verifythat the electionofficialarea 20 was initialled by the election official to ensure the ballot was properly issuedto an eligiblevoter, and perform a pixel count of each voter selection area 18 on the ballot 17. In addition, as further examples of such processing results (not shown), dependingon the pixel connt of each markingbox, the mark is eitherclassifiedas a 'vote', a 'non-vote', or an 'ambiguous marking'. These classificationsare based on the total pixel counts of the marking areas ("marking area" or "marking areas" refer for example to voter selection areas 18 or election official areas 20 and other similar areas) of the ballot, and the corresponding pixel levels defined by the election officialsat a given time prior to the election. The electionofficialsclearlydefine,in pixels,the minimumpixel count that is to be classifiedas a 'vote', the maximumpixel count (if any) that is to be defined as a definite 'non-vote', and a range of pixels in between those values that will constitute an 'ambiguous marking'. These pixel values are loaded on each tabulation unit 10 After calculating these pixel values for each marking area, the tabulation unit 10 will return the ballot (by operationof the paper feed mechanism 26) to the voter with an appropriateerror message for any of the followingscenarios(as a particularillustrationof the present invention): any marking area was classifiedas an 'ambigious marking'; too many markingareas in one categorywere classifiedas 'votes' resulting in an "over-vote"; no marking areas in one or more categorywere classified as 'non-votes', resulting in a blank ballot; and/or

US 9,710,988 B2 7

8

no pixel count was recorded in the election officialarea 20, indicating that the ballot was not initialed by an election worker. It shouldbe understoodthat the tabulationunit 10 can be programmedto allow a voter to verify its ballot in the case of over-voted or blank ballots, thus preserving the voter's right to cast an over-votedor blank ballot. If the ballot was not returned to the voter for any of the abovereasons,the computer(not shown)determinesthat all of the votes pertainingto the particularballot 17 have been cast by that ballot, and appropriately increments a main counter (not shown) that forms part of the memory (not shown),in a particularembodimentof the present invention. In addition, once the determinationof the total votes for that ballot 17 has been made, by further operation of the ballot processing application28, the computer (not shown) appends a footer 30 to the saved ballot image (not shown), which contains all of the processingresults (not shown) for that specificballot 17, as shownill FIG. 4. The footer30 can also be understoodas a "fingerprint"or a "vote stamp" that is attachedto each image (not shown).The presentinvention thereby permits the creation of one random, clear, image of every ballot 17 along with all the informationthe tabulation unit 10 has compiledfor that particularballot 17 that can be later viewed by any software image viewer (not shown). Upon successful completion of the ballot processing, in accordance with the present invention, the ballot 17 is passed the rest of the way through the tabulation unit 10, where it drops into the ballot box 12, where it mixes freely with the otherballotsto preservethe anonymousorderingof the ballots (as shown in FIG. 2). In a particular aspect of the present invention, means is provided for voting by an individualwith a challenge such as visual impairment, illiteracy, inability to read the language in which the ballot is written, physically impairment with only limited motor abilities, etc. In such case, in accordancewith another aspect of the method of the present invention, the election official initializes an 'audio ballot' (not shown)which is administereddirectlyby the tabulation unit 10 (in one particularembodimentthereof).By operation of another aspect of the ballot processingapplication28 of the present invention, the election official initializes, or activates, the audio ballot by using an electronic key (or some other suitable mechanism), and ushers the voter to a voting area wherethe voter can put on the ear phones 14 and take hold of the handheld pendant that includes the manual trigger selector 16. The audio ballot (not shown) is best understood as an audio version of the ballot 17, which is stored to the memory (not shown) associated with the computer (not shown). The voter negotiates through the ballot 17 through the audio interface and makes candidate selections via the manual trigger 16. Every time the voter makes a selection, and everytime a voter completesvoting for every individual race, the computer(not shown)asks the voter to confirmthe choices after it is audiblyreviewed.Upon completionof the audio ballot, a final review is read back to the voter, and the voter is askedone more time to confirmthe selectedchoices. This audio clip of this review, along with the subsequent sound of the voter confirmingthe review, is recorded and randomly stored in memory (not shown)by operationof the ballot processingapplication28. In addition,once the voter is finishedwith this final review, the election officialgives the voter one last chance to cancel the ballot and start over. Once this option has been denied by the voter, the election officialuses their electronickey to cast the audio ballot. By doing so,the votes fromthat audioballot are appendedto the

ongoing tally of the tabulation unit 10 (as particularized above), the sound bites are saved to the memory, and optionally a printer linked to the computer (not shown) prints the votes summaryof the audiovote onto a paper chit. The chit then falls into the ballot box 12 in orderto maintain secrecyor is printedin encryptedformfor audioplaybackon either the tabulationunit 10 or external device. This aspect of the method of the present invention is also illustrated in FIG. 1. At the end of voting, the tabulation unit 10 includes an ongoing tally for all the ballots that passed through that particular machine, and those results are transmitted (by various known methods and technologies) to a central location which includes a central election database 32, as illustrated in FIG. 5. The various election results from the varioustabulationunits 10 are summedtogetherin a mauner that is known and storedto the central election database32, in a marmerthat is known. It should be understood that one aspect of the present inventionis a systemconsistingof the tabulationunit 10 of the present invention,and which includesthe computer(not shown) and the ballot processing application 28. Another aspect of the present inventionis a further systemconsisting of a plurality of tabulation units 10 linked to a central computer (not shown), the central computer (not shown) being linked to the central election database 32. The systemand computerprogram aspects of the present inventioncan be providedin a mannerthat is knownto those skilled in the art. Particularhardware and software components are describedbelow for illustrationpurposes. TechnicalDescriptionof the TabulatorHardware The tabulation unit 10 is generally a self-containedunit made up of severaldifferentsubcomponents.The following is a list and description of representative subcomponents. Motherboard In one particularaspectof the presentinvention,the ballot processingmechanism 22 includes a motherboardthat provides the functional block of the tabulation unit 10, and which controls electronic processing of the tabulator process. The motherboard (not shown) preferably includes severalinterfaces,includingan interfacewith the paper feed mechanism 26, an audio/general purpose input output (GPIO) interface, a scanning card and image heads (associatedwith the digitalscanningdevice24),the power supply board, a thermal printer,and the user interface34 illustrated in FIG. 2. The motherboard(not shown)includesthe computer(not shown) consisting of a known microprocessor.The motherboard (not shown) along with its processor (not shown) can be made to operate with a variety of softwareoperating systems, but currently is utilizing either WINDOWS™ CE.Net or uClinux™. The processor (not shown) may consist of an x86 INTEL™ based processor (for the Windows CE.Net version) or, for example, a MOTOROLA COLDFIRE™processor (for the nClinux™ version). The motherboard (not shown) has various sources of memory including, for example, onboard flash memory where the operating system kernel resides, as well as various hard drive options including an on-boardATAflash drive and a removable compact flash drive. Paper Feed Mechanism The paper feedmechanism26 is that part of the tabulation unit 10 that physically controls the movement of the paper ballot throughthe tabulationunit 10. In a particularembodiment of the present invention,the paper feed mechanism26 is of a very similar in construction to todays modem sheet-fed scanning systems (or sheet-fed fax machines) in

10

15

20

25

30

35

40

45

50

55

60

65

US 9,710,988 B2 9

10

that it incorporatesa systemof rubberizedwheels connected to a stepper motor that are responsible for advancing the paper either forward or backward through the paper feed mechanism. The paper feed mechanism 26 also houses a object sensor system which is a series of LED/photodetectors used to detect a sheet of paper that has been inserted into the opening slot of the paper feed mechanism 26. Also included on the paper feed mechanism 26 is a double paper sensor, which is a commercially available module that is capable of detecting more than one sheet of paper. Lastly,the paper feed mechanism26 also houses two separatelypackaged contact image sensor (CIS) units (one above the paper feed plane, and one below allowing for duplex imaging), which are commerciallyavailable and in large scale use in commercial fax and scanning machines. The paper feed mechanism26 is preferablydesignedin such a way as to allow the rubber wheels to advance and reverse the paper through the paper feed mechanismand across the mounted CIS units in such a manner as to allow no more than 7 degrees of paper skew, and in such a manner as to ensure that the paper does not bind with any part of the tabulator, thus ensuring that the paper does not jam inside the tabulator. The opening slot 36 of the paper feed mechanism 26 is extendedout through a front slot 38 of the ballot box 12 for the insertion of the paper, and a rear slot is 40 extended throughthe rear wall of the paper feed mechanismensuring that the paper is ableto freely exit the paper feed mechanism and fall directly into the ballot box 12. Scanning Card and Image Heads The digital scanning device 24 of the present invention preferably includes a series of imaging heads or modules (not shown), which can be either be of the contact image sensor (CIS) or charge coupled device (CCD) type. When prompted, each imagingmodule (either CIS or CCD) takes a one dimensionalimage of the ballot paper that is in direct contact with the module, producing a series of black and white or grey-scalepixels that represent the image for that particular line of the paper. By couplingthis process with a single line advance of the steppermotor (i.e. advancingthe paper by one line) and repeating, a very high quality two dimensional image of the entire sheet of paper is created. The imaging modules interface directly with the image controller.In one form of the tabulation unit 10, this controller is in the form of a separatelymountedcustomprinted circuit board (PCB). This board contains an application specific integrated circuit (ASIC) chip that has been designedfor specificuse in scanningapplications.This chip is commerciallyavailable and is in wide use in commercial scanningproducts.In its other form, this imaging controller resides directlyon the Coldfireprocessorwhich is locatedon the main motherboard.In this version of the controller,the softwarethat is encoded on the ASIC chip is ported directly onto the main processor, thus eliminating the need for a separate PCB. In either case, the imaging controller is one particular aspect of the ballot processing application 28 of the present invention. The function of this imaging controller is to control the feeding and scanningof each ballot 17. When 'armed' (i.e. the main tabulator applicationsendsthe signal to the image controllerto be ready to accept and scan ballots) the image controller monitors the paper sensor hardware. When a ballot 17 is insertedinto the openingof the tabulator (i.e. the front of the paper feed mechanism system 26), the paper sense hardware trips, and sends the signal to the image controllerto start the scan routine. In this scan routine, the image controllerrepeatedlytriggersthe imagingmodulesto

scan a line and then advancesthe paper feedingwheels (i.e. via the stepper motor) and repeats until the paper has been completelyscanned.Whenthe paper is completelyscanned, two complete images are sent by the imaging controller to the main tabulator application for processing. During this time, it is important to note that the paper has not been completely fed through the paper feed mechanism. Even though the ballot has completelypassed over each imaging head, the bottompiece of the ballot is still being held by the rear rubberizedwheels. It is at this point where the imaging controller awaits instructions from the main processor or computer (not shown) for its next instruction. This next instructioncanbe eitherto completelyreturnthe ballotto the voter (i.e. reversing the direction of the stepper motor allowing the rubberized wheels to move the paper back through the machine and out the front slot) or to accept the ballot (i.e. forwardingthe stepper motor a predefinednumber of steps to allow the ballot to continuethrough past the rear slot 40 of the tabulator, allowing the ballot to fall into the ballot box). After this instruction, the controller will 're-arm' itself in preparation for the next paper ballot until such lime as the main tabulatorapplicationtells it to turn off. Power Supply The power supply is generally a separate PCB that convertsthe incomingpower voltage (either 120VAC,240 VAC, or 30 VDC) to a DC voltage that is required by the other PCB's in the tabulator.Specifically,-5VDC, +5VDC, +12VDCand -12VDC are the outputs of this PCB, and are used to power the motherboard, the scanning board, the user-interfacemodule, and the thermal printer. Audio/GPIO Card The audio/GPIO card is either in the form of a separate PCB, or built-in on the motherboard. Its function is to provide an audio output and a GPIO input to the motherboard. The card interfaces to two mini 3.5 mm connectors that are mounted on the rear plate of the tabulator; one is where the earphones 14 plug in, and the other is where the handheldpendantthat housesthe manualtrigger 16 plugs in. This card allowsthe main applicationto play various sound files via the earphone jack, and accept user input via the handheld pendant. Thermal Printer This is a commercially available thermal printer which interfaces directly with the motherboard, in one particular embodimentof the present inventionmentionedabove. The printer is mounted in such a way that the paper output is aligned with a slot in the tabulator cover, allowing the printed piece of paper to feed throughthe tabulator cover to the outside. When sent a print job by the motherboard,the printer produces a printout on a thermal paper roll. User Interface The user interface34 is a knowninterfacedevicemounted to the tabulator cover, which possessesboth an information display screen and a method of user input. In a preferred embodimentof the presentinvention,there are currentlytwo different types of this module. The first is a TFT display screencoupledto a resistivetouch-screeninput device.Both are commerciallyavailable pieces of hardware. The second type of user-interface module available is a module that possesses an LCD display screen with several keypad buttons. Both modules interfacedirectlywith the motherboard, and are controlledby the ballot processingapplication28. It is through this device that election officials are able to interface with the tabulator. This interface is used for a numberof situations,includingenteringadministrativepasswords, settingup audioballots,finalcastingof audioballots, printing various types of election reports, accepting or

10

15

20

25

30

35

40

45

50

55

60

65

US 9,710,988 B2 11

12

returning over-voted or blank ballots, etc., which constitute any documents (i.e ballots). The digital scanning device 24 further functions of the ballot processing application 28. is initially in this inactive state. It will be in this state until Chassis and Exterior Package the initialization routine is complete and everything is okay. The chassis is made up of a metal base and metal rear It can be put back in this inactive state by the application panel which is mounted to the base at 90 degrees. The power layer. supply PCB, the scauner PCB, the motherboard PCB, therThe overall speed of a scan from once the paper is sensed mal printer and audio/GPIO board are all mounted to this to when the driver notifies the application layer that a ballot metal chassis. The paper feed mechanism is then mounted to 17 has been scanned is generally under one (1) second, using the base as well, but sits above the above other modules. The existing technology, which is suitable for the purpose of the user interface module is fastened to the paper feed mecha- 10 present invention. nism cover (which can be either plastic or metal) which in The driver described will generally allow the scan of any turn fastens to the metal base and rear plates via tampersized document up to a maximum of 21 inches long. proof security screws (i.e. torx screws). Description of Random Auditing Process Description of Scauning Software As described above, once the vote has been completed, As mentioned above, one aspect of the ballot processing 15 the vote result files are transferred to the central election application 28 of the present invention is an application database 32. Election officials can now perform a random layer that calls the scanning function that points to a matrix. auditing process, in accordance with another aspect of the invention, to ensure that the tabulation units 10 have funcWhen a ballot 17 is scanned a driver of the ballot processing application 28 is responsible to copy the digital scan of the tioned correctly (and also to verify the integrity of the vote document to a specified location on the memory (not 20 overall). This can be in the form of a spot check, a structured shown). check to gain a certain confidence rate in the tabulator When the scanner is enabled it is in ready/active mode and integrity, or a complete check of every ballot cast. will await the insertion of a ballot 17 in the scanner. In this For the sake of clarity, once the vote result files from each mode of operation, the digital scanning device 24 will await tabulation unit 10 have been transferred to the central the insertion of a ballot to begin the scan. The operation of 25 computer (not shown), there will be one multimedia file for the scanner and driver during the scanning process is outevery ballot cast during the election, complete with the lined below: corresponding processing results (not shown) or vote 1. When a ballot 17 is inserted into the digital scanning stamped 'fingerprint' information. In the case of paper device 24, the front Object Sensor will send a signal to ballots, there is one image file for every ballot cast, and in the scanning board which will notify the driver. If the 30 the case of audio ballots, there will be one sound file for driver is in active mode, it will start scanning the every audio ballot cast. Depending on the auditing strategy document. used, the auditing committee (for example) can have any 2. At the start of each line scan, the driver will send a 'start number of terminals accessing the ballot files from the pulse signal' to the CIS module (where a CIS module central election database 32, as illustrated in FIG. 5. Each is used, as explained above). This prompts the CIS 35 auditor working on a separate computer can open as many module w place an analog output voltage on its Vout ballot files as needed. pin that is representative of the darkness of the first Audit Process pixel. This is followed by a sequential series of Vout The process that allows auditors to verify that any given voltages for each of the pixels in the line. These signals ballot was tabulated properly is as follows. As explained are spaced by the internal CIS clock, which is set by the 40 above, a complete image of each ballot 17 scanned is saved driver. As the digital scanning device 24 is going to the ballot file. In addition, the fingerprint of that ballot through its line scan, the scanning board will convert (which contains all the pertinent information that the tabueach voltage signal to a digital signal, in a manner that lator obtained from that ballot, including pixel values for is known. The threshold value of this quantization is each marking area as well as which candidates where voted configurable. 45 for) is vote stamped below that image by including same in 3. The resultant digitized image (not shown), again, is the footer 30. By simply opening any given ballot file on stored in the predefined location of the memory (not their computer terminal, the election auditor is able to shown). compare what the voter markings to the tabulator interpre4. When the back Object Sensor is triggered, the scanning tation. In the case of the audio ballot, the election auditor can board will notify the driver. At this point the driver will 50 compare the sound file that the voter reviewed and connotify the application layer (via a function call) that the firmed with what the tabulator recorded for that ballot. Thus document has been scanned, imaged, and is sitting in by simply opening one of these multimedia files, the auditor matrix form in memory. The driver will also, via this can verify whether the tabulation unit 10 made an error in function call, let the application know the size of the tabulating that particular vote. matrix. Here, the driver will be put in a wait state. It 55 In FIG. 6, we see an example of a ballot that was will wait for instructions from the application layer. incorrectly tabulated. It is clear that the voter intended to vote for the write-in candidate 'Nick Salvatore', but the 5. A driver defined API is provided so that the application can then instruct the driver to either pass the document tabulator registered a vote for Pierre E. Trudeau. In essence, completely through (forward) the digital scanning to verify that one ballot has been tabulated correctly takes device 24, or completely reverse the direction of the 60 only a few seconds. stepper motor, returning the entire document back Types of Auditing Strategies through the front of the unit. The main advantage of this type of fingerprinting and Also if the driver detects any errors in the document (i.e. corresponding audit trail is that it easily enables virtually double sheet, paper jam, etc.) then it will reverse the any auditing strategy. On the one extreme, perhaps only one document and notify the application layer of the error. 65 or two ballot files from each tabulation unit 10 would be During this disabled state the driver will disregard the checked to verify that the tabulator fingerprint matched the beginning of document signal from the device and not scan original ballot information. This would provide some degree

US 9,710,988 B2 13

14

of a sanity check against a systematicproblem,but depending on the number of ballots cast at each tabulator, would generatea fairly low confidencerate on the auditingprocess. On the other extreme, would be a 100% confidencerate on the audit. For this, every single ballot file would be openedto confirmthat the tabulator fingerprintmatched the original ballot information. However, depending on how many ballots had been cast during the election and how many audit workerswere availableto carry out the auditing, this process could be too slow. It should be understoodthat the present inventioncan be used in many voting scenarios,includingfor example shareholder votes and the like. Specificaspects of the hardware and software described can be modified without departing from the system and computer program aspects of the present invention. What is claimed is: 1. A method for determiningvotes recorded on a votermarked paper ballot, comprising: receiving, at a ballot processing computer,optical image data comprising an optical image of a voter-marked paper ballot, the voter-marked paper ballot including voter selection areas; performing a pixel determination of one or more voter selection areas, the pixel determination identifying pixels in the opticalimagethat containa voter marking; determiningthat a first subset of the one or more voter selection areas have been selected based on the pixel determinationwhen a pixel count of one or more voter selectionareas exceeda predefinedfirstthresholdvalue for determininga specificvoter selectionarea has been selected; determiningthat a second subsetof the one or more voter selection areas have not been selected when a pixel count of one or more voter selectionareas fall below a predefined second threshold value for determining a specificvoter selection area has not been selected; determiningthat a third subset of the one or more voter selection areas includes at least one ambiguous mark when a pixel count of one or more voter selectionareas fall between the predefined first threshold value and second threshold value; and outputtingan indicationthat one or more voter selection areas are ambiguous when it is determined that the voter-markedpaperballot includesat least one ambiguous mark. 2. The method of claim 1, wherein the outputting an indication comprises: identifyingmarks on the voter-markedpaper ballot determined to be ambiguousmarks. 3. The method of claim 2, further comprising: routing the voter-markedpaper ballot to a first location when it is determined that the voter-marked paper ballot does not include at least one ambiguous mark; and routingthe voter-markedpaperballot to a secondlocation when it is determined that the voter-marked paper ballot does include at least one ambiguousmark. 4. The method of claim 3, wherein the first location comprises a ballot box. 5. The method of claim 3, wherein routing the votermarked paper ballot to a second location comprises: returning the voter-markedpaper ballot to the voter. 6. The method of claim 1, further comprising: determiningone or morevotes based on one or morevoter selectionareas that have been determinedto have been selected.

7. The method of claim 6, further comprising: identifyingan overvotewhen votes for an identifiedrace exceed an allowed number of votes for the identified race. 8. The method of claim 1, further comprising: performing a pixel determinationof each voter selection area, the pixel determinationidentifying pixels in the optical image that comprise a voter marking. 9. The method of claim 8, wherein the determiningsteps are performed based on the pixel determination for each voter selection area. 10. The method of claim 8, wherein the predefined first threshold value comprises a minimum number of pixels to be classifiedas a vote. 11.The method of claim8, whereinthe predefinedsecond threshold value comprisesa maximumnumber of pixels to be classifiedas a non-vote. 12. The method of claim 11, wherein the maximum numberof pixels to be classifiedas a non-voteis zeropixels. 13. The method of claim 1, further comprising: determiningone or morevotes based on one or morevoter selectionareas that have been determinedto have been selected; and generatinga vote stampcomprisingthe one or more votes. 14. The method of claim 13, further comprising: appendingan image of the vote stampto the opticalimage of the voter-markedpaper ballot. 15. A system for determiningvotes recorded on a votermarked paper ballot, comprising: a processor; and a memory in electronic communicationwith the processor,whereinthe memory stores executableinstructions that when executedby the processor cause the processor to perform operations comprising: receiving optical image data comprising an optical image of a voter-marked paper ballot, the votermarked paper ballot includingvoter selectionareas; performinga pixel determinationof one or more voter selection areas, the pixel determination identifying pixels in the optical image that contain a voter marking; determiningthat a first subset of the one or more voter selectionareas have been selectedbased on the pixel determination when a pixel count of one or more voter selectionareas exceeda predefinedfirstthreshold value for determining a specificvoter selection area has been selected; determining that a second subset of the one or more voter selectionareas have not been selectedwhen a pixel count of one or more voter selectionareas fall below a predefinedsecondthresholdvalue for determining a specificvoter selection area has not been selected; determiningthat a third subsetof the one or more voter selectionareas includesat least one ambiguousmark when a pixel count of one or more voter selection areas fallbetweenthe predefinedfirstthresholdvalue and second threshold value; and outputting an indication that one or more voter selection areas are ambiguouswhen it is determinedthat the voter-markedpaper ballot includes at least one ambiguousmark. 16. The system of claim 15, wherein the memory stores further executable instructions that when executed by the processor cause the processor to perform further operations comprising:

10

15

20

25

30

35

40

45

50

55

60

65

US 9,710,988 B2 15 determiningone or morevotes based on one or morevoter selectionareas that have been determinedto have been selected. 17. The system of claim 15, wherein the memory stores further executable instructions that when executed by the processor cause the processorto perform further operations comprising: performing a pixel determinationof each voter selection area, the pixel determinationidentifying pixels in the optical image that comprise a voter marking. 18. The system of claim 15, wherein the predefinedfirst threshold value comprises a minimum number of pixels to be classifiedas a vote. 19. The system of claim 15, wherein the predefined second threshold value comprises a maximum number of pixels to be classifiedas a non-vote. 20. The system of claim 19, wherein the maximum number of pixels to be classifiedas a non-voteis zero pixels.

* * * * *

16

10

15

111111

(12)

United States Patent

1111111111111111111111111111111111111111111111111111111111111 US009870666B2

Patent No.: US 9,870,666 B2 (45) Date of Patent: *Jan. 16,2018 (10)

Poulos et al. (54) SYSTEM, METHOD AND COMPUTER PROGRAM FOR VOTE TABULATIONWITH AN ELECTRONIC AUDIT TRAIL

(58) Field of Classification Search CPC G07C 13/00; G06Q 20/382; G06Q 20/401 USPC 705/12 See application file for complete search history.

(75) Inventors: John Poulos, Toronto (CA); James Hoover, Montreal (CA); Nick Ikonomakis, Toronto (CA); Goran Obradovlc, Toronto (CA)

(56)

References Cited U.S. PATENTDOCUMENTS 6,250,548 Bl

(73) Assignee: Dominion Voting Systems, Inc., Denver, CO (US)

200210091673 Al 200210143610 Al

*

6/2001 McClureet al. 7/2002 Seibel et al. 1012002 Munyer

G06Q 10/10 705/12

( *)

Notice:

Subjectto any disclaimer,the term of this patent is extended or adjusted under 35 U.S.C. 154(b)by 916 days.

(Continued) OTHER PUBLICATIONS

This patent is subject to a terminal disclaimer.

Source Code for ES&S Model 100 Precinct Ballot Counter (formerly AIS Precinct Ballot Counter), Module votecnt.C, Revision 2.62, 3 pages (redacted),Apr. 3, 2000.

(21) Appl. No.: 13/463,536 (22) Filed:

(Continued)

May 3,2012

(65)

Primary Examiner - Chrystina E Zelaskiewicz (74) Attorney, Agent, or Firm - Holland & Hart LLP

Prior Publication Data US 2012/0232963Al

Sep. 13, 2012

(57)

Related U.S. Application Data (63) Continuationof applicationNo. 11/121,997,filed on May 5, 2005, now Pat. No. 8,195,505. (30)

Foreign Application Priority Data

May 5, 2004

(CA)

ABSTRACT

A system, method and computer program for tabulating votes and creating an audit trail is provided. A ballot processing device may include a paper feed mechanism, a computer, a ballot processing application loaded on the computer,and a digital scanning device linked to the computer. The ballot processing application may process the digital image to establish a series of processing results definningone or more votingresults for the paperballot, and also an audit trail. The ballot processing application may process the digital image to define the voting results based on criteria established by election officials, including ambiguous mark criteria. The audit trail enables election officials to verify that particular paper ballots have been processed correctly in accordancewith these criteria.

2466466

(51) Int. Cl. G07C 13/00 (2006.01) G06Q 20/38 (2012.01) G06Q 20/40 (2012.01) (52) U.S. Cl. CPC . G07C 13/00 (2013.01); G06Q 20/382 (2013.01); G06Q 20/401 (2013.01)

18 Claims, 7 Drawing Sheets

3

;r

.a

j -a~ i ~ j

~

•

..

~ ea ~~ '" ea ea

II ~ ~ ~

~ ~ ~ ~~ g

I

0 0 0 0 0 0

~

i

"" ea ea ea ea ea

~17

US 9,870,666 B2 Page 2 (56)

References

Cited

U.S. PATENT DOCUMENTS 2003/0052981

Al

2003/0173404 2007/0170253

Al Al

*

3/2003

Kakarala

*

912003 7/2007

Chung et al. Chung

G06T 3/4007 348/272

G06K 7/10346 235/386

OTHER PUBLICATIONS Election Systems and Software, Inc., ES&S Model 100 Precinct Ballot Counter Operator's Manual, 101 pages, Apr. 2000. Requisition by the Examiner for Canadian Application No. 2,466,466dated Jul. 8, 2015, from the CanadianIntellectualProperty Office. Requisition by the Examiner for Canadian Application No. 2,466,466 dated Aug. 3, 2015, from the Canadian Intellectual Property Office.

* cited by examiner

u.s.

Patent

Jao.16,2018

US 9,870,666 B2

Sheet 1 of 7

I

ToFig.1b

Audio

Print the required ballot 'on-demand' on the local printer and issue to voter

Elettion official to place markedballot is put into the 'Spoiled Ballots' envelope, Voteris issued a new ballot and is reinstructed on ballot marking

Ballot feeds completely through the tabulator and falls into the ballot box Tabulator is ready for next paper ballot

FIG.1a

u.s.

Patent

FromFig.1a

Jao.16,2018

Sheet 2 of 7

US 9,870,666 B2

Voter is escorted to the audio voting area where he is given a set of ear hones and a handheld pendant thumb switch Once the voter is set up, the election official selects the proper audio ballot on the tabulator to initialize the voting session Tlie voting session begins from the beginning and the voter 'navigates' through the ballot selecting as many candidates as desired At the end of the audio voting process, the voter is prompted to review the selected choices one last time and to confirm the selections

The ballot is cancelled and the voter returns to the start of the ballot to begin again

Yes The tabulator makes an audible sound to alert the election official thai the audio vote has been completed No The election official assist the voter in leaving the audio voting area

The election official prompts the tabulator to accept the ballot, and a paper reciept is printed and falls into the ballot box

FIG.1b

Tabulator is ready for the next ballot

u.s. Patent

US 9,870,666 B2

Sheet 3 of 7

Jao.16,2018

10 24

14

~

.,0 f}--16

----_ ....., Auxiliary Ballot Box

Tabulator Ballot Box

12

FIG. 2

u.s. Patent

US 9,870,666 B2

Sheet 4 of 7

Jan. 16,2018

~ fZiZI

City of Anywhere

I
General Municipal Election January 11891

IZZI

fZiZI Tovote, completely fill inthe box(esl

0next to your choice(s). like this:

•

::~==============~==============~

I
BALLOT FOR i;EOFFI:E

IZZI

_liJronlyONE (l)candidaIe

18~L-- I
11Robert L ..2lJean

MAYOR

BORDON-EJ

CARTIER

~

.)Samuelde

CHAMPLAIN

I
S)James

COOK

~

6) Lei!

ERIKSSON

fZiZI

7)John

FRANKLIN

I
81 Henry

~

9)Jake

~

10)WIlliamLM.

~

12)John A.

~

.,.,..,arrtgtt

Carddafel 00 Carddafe 2 • 0 Cancidolo3

HUDSON JOLIET KING LAURIER MacDONALD PEARSON

14) LouisS.

SAINT·LAURENT

IZZ?I15)PierreE. IZZ?III ~ L.....

TRUDEAU

0 0 0 0 0 0 0 0 0 0 0 0

o.

...I;t- '\,~

CABOT-£]

31Jacques

IZ1ZI 13) Lester B.

COUNCILLOR

IAJIe bONE (I) 151d1oice & Qt.E (112nd cIIoIce,as

I
IZ1ZI 11)Wilfrid

BALlOHOR g:EOFACE

1)MilesJ. 2)MenelausP. 3) Clause

0 000000 10__J

FIG. 3

MASSEY O'DANIEL VON ESPIE

0 0 0 0 DD

Il'ZJ

u.s. Patent

Jao.16,2018

US 9,870,666 B2

Sheet 5 of 7

IZi2I

January

ra.zJ To vole, completely fill in the bOlles)

ra.zJ

18

Ilm

BAllOTFOR THEOFFICE Of

Ilm

VOleloronlyONE(11candidale

't--IZZI

I'Wi ~

MAYOR

1) RobertL

o

nell to your

31Jacques

7) John

CARTIER CHAMPLAIN COOK ERIKSSON FRANKLIN HUDSON JOLIET KING

11)Wilfrid

IZZI 12)John A. IZZI 13)lester B. IZZI

14)louis S.

Iz:zI

15)PieneE.

IZZI

I

lAURIER MacDONALD PEARSON SAINT·lAURENT TRUDEAU

f\licl< Salvator

IZZI

~

•

choicelsl,like Ihis:

BALlOTFOA THE OFFICE OF

COUNCILLOR CandidiltBl 00 CandidalaZ • [J Candidole3

o.

...~ <#

CABOT'D

ra.zJ 8) Henry IZZI 9)Jake IZZI 10)William l.M.

~

11891

IhlIaIirCNE(IJl~_ & ONE(1) ind_,as _alright

BORDON

.,aJean

IZZI 4) Samuelde IZZI 5) James IZZI 6)LeH

~

1i-s.1

General Municipal Election

IZZI

ra.zJ

ORO

City of Anywhere

IZi2I

0 0 0 0 0 0 0 0 0 0 0 0 0

,m

1) MilesJ. 2) MenelausP.

O'DANIEL

3)Clau&e

VON ESPIE

~ ~ .!::! ~ ~ iii" ~ -"! -< ~ a: ~ s

..

.!!!

s:: .-

-=

J

~ I.'iZI I'Zi'J I'Zi'J I.'iZI I'Zi'J I.'iZI £iZI I.'iZI £iZI £iZI

0

£iZI

0 0 0

I!:ZI

MASSEY.

•

£iZI I!:ZI

i'fO ....~ &) ,..~

if

~~

I!:ZI

.ao fie

£iZI

:i!

llZI

m m

r;zJ

m;Q

'"

B

""'

<> :z ..... C> n <> In z: ::a

<> (I)

....

... ...<>'" m

e-c

'" c

== m ::a -<

(")

..

;r::

c: Z c:>

m

3:;

r-

,...z: on,... n

C :0

I!:ZI

~ ..>

I!:ZI

i

'"'"

0

r;zJ

::u

I!:ZI

> :z

I!:ZI

m

3:

IliPJIll10MO

I!:ZI I!:ZI

m m m m m m m m m

ORO: 2109 Office of Ma~or: 0.0.0,1,0,0.0,0,1,0,0,0,0,1,0,780 Vote for: Nic Salvator " Office of Councillor: (800.0), (1,790), (0,0) Vote for: Miles J. MASSEY (1st). Menelaus P. O'DANIEL (2nd) Office of ALDERMAN: Vote for: OVER·VOTE

9

m m m m m m m m

m

~

2,812.2.823,788.774

FIG. 4

, ( 30

17

u.s. Patent

Jao.16,2018

US 9,870,666 B2

Sheet 6 of 7

Data Card from Data Card from Data Card from Tabulator 1 Tabulator 3 Tabulator X Data Card from Ie::] Data Card from Tabulator 4 ~ l.m;JJ Tabulator 2

o

0 lItLl

~

~

,

Computer Central Election Database

32

I

Auditing PC Auditing PC Auditing PC Auditing PC Station #1 Station #2 Station #3 Station #Y

FIG. 5

I

u.s. Patent

~

US 9,870,666 B2

Sheet 7 of 7

Jan. 16,2018

~

m m m m m m m

~

m:I

January 1 1891

m:I

IZ:ZI

TItYote, completelyfill in the box(es)

IZ!ZI

BAU.OTFOR THEOFFICE OF

IZ!ZI

_'or~ON;(l)G3IIcidate

IZ!ZI

1)RobertL

IZ!ZI

2)Jean

IZ:ZI

3) Jacques

IZ!ZI IZ:ZI

5) James

IZ!ZI IZ!ZI

MAYOR

61 Lei! 7)John 8) Henry

IZ!ZI

10)William L.M.

IZ!ZI

11)Wilfrid

IZ!ZI

12j'JohnA.

9)Jalce

m:I 13)LesterB. m:I ''') LoulsS. 15} PierreE.

IZ!ZI

~

o next to your choice(s),like this: BALLOT FOR THE OFFICE _ for ONE (1)151 dIoice & ONEIllblchaice,'. _alright

NICK

Salvator

2) Menel_P.

Al' '" .o!

VON ESPIE %

-Ii

S' :r

m

0

... ..."'" z: r-' Q

G> 0 3: m

-< '" '"

I:>

•••

e ~

l!I: 0

z: _,

'"

<'

<'> rc:: %

C>

lii: »0 Z m

...1iI- ...~

IZ'J

rzza

0 .... m~

,.. .., <'> '" '"

if i

fiZI IZ'J fiZI Il'ZI I?Zl IZ'J

~ rC

m ::c

s: ~ :z:

OIJlO

fiZI

{';ZI

Im

rzza IZ'J

III 0 0• 0 0

f~~~ 501= ... ~9 ....

... n

IZ'J

rzza

'" "' "' £; ;u

rzza rzza

rzza

~ .e ~ -!:=! ~m ", ~'" j ~ g; ~ a: li:

0 en m

rzza

CancIdotel00

O'DANIEL

3) Clause

I?Zl

CaIIidaIe2 .' 0 Ca1
MASSEY

1)MilesJ.

s

•

COUNCILLOR

OF

CABOT

4) Samuel de

IZ!ZI

0 0 CARTIER 0 CHAMPLAIN 0 COOK 0 ERIKSSON 0 FRANKLIN 0 HUDSON 0 JOLIET 0 KING 0 LAURIER 0 MacDONALD 0 PEARSON 0 SAINT·LAURENT 0 TRUDEAU 0 BORDON

IZ!ZI IZ!ZI

IZ!ZI

li·s.1

General Municipal Election

m:I IZ:ZI

ORO

City of Anywhere

IZ:ZI

~ ~

Il'Z3 C'ZI I?Zl Il'Z3 Il'Z3 Il'Z3

m m m m m m m m m

ORO:2109 Office of Mallor: 0,0,0,1,0,0,0,0,1,0,0,0,0,1,0,780 Vote for: Pierre E. TRUDeau Office of Councillor: (800,0), (1,7901, (0,0) Vote (or: Miles J. MASSEY (1st). Menelaus p, O'OANIEL (2nd) Office of ALDERMAN:2,812.2,823,788,726 Vote for: OVER-VOTE

FIG. 6

\

~17

US 9,870,666 B2 1

2

SYSTEM, METHOD AND COMPUTER PROGRAM FOR VOTE TABULATIONWITH AN ELECTRONIC AUDIT TRAIL

One of the challenges in permitting efficientverification of voting results, and also demonstrationof the integrity of the vote, are the subjectiveelements generally involved in determiningvoter intent. In accordance with numerous prior art voting technologies, the assessmentof what constitutes a vote in favour of a particularcandidateor a particular answerto a questionis determinedby subjectiveassessment. This is best illustrated in the context of a manual hand count. Historically, voters would select their candidates using pen and paper, making a mark (an 'X' for example) beside the candidate(s) of their choice. This paper ballot would then be manually counted by election officialswith candidatesand their agents allowedto scrutinizethe operation. However, by the very nature of elections, different parties involvedwith electionshave very differentinterests. A set of candidates and their agents rarely share common motivation and accordingly the interpretation process of what constitutes a mark on a ballot has always been prone to inconsistencies.This fact generally makes it difficultto achieve the same tally when counting the same ballots multipletimes, as there is no assurancethat any givenballot wouldbe interpretedthe sameway by everyinterestedparty. The reason for this is straightforward:often the voter would make a mark that couldbe deemedas ambiguousby anyone of the people involved with the manual ballot count (i.e. electionofficialor candidateagent).Whatmay be obviousto one agent may not be obviousto another agent. In addition, many ballots were deemed 'spoiled' or 'blank' because the voter (eitherintentionallyor unintentionally)made too many or too few marks on the ballot. The apparent lack of objectivity was addressed in a numberof prior art systems.The MechanicalBased Systems and Optical-MarkBased Systemseither incrementeda particular vote or not based on objective standards. The Mechanical Based Systems have been deemed susceptible to inconsistencies in well publicized cases. For example,somemechanicalpunch systemshave been shown to leave cut-out 'chads' partially attached, leading to disputes surroundingelection results. Also, in the case of Mechanical Based Systems and Optical-Mark Based Systems, the objective standards for what constitutes an indication of voter intention are generally set by the manufacturerof the voting device and have not been adaptableto conformto objective standardsdetermined by election officials for a particular vote. For example, in Optical-Mark Based Systems an integration thresholdis usually definedto establishhow dark a mark has to be in order to constitutea mark recognizedby the system as being such. This integrationthreshold is permanentlyset by the manufacturer. These concerns have hindered the ability to verify vote results,and have generallyhad a negative impacton impressions of vote integrity. To some extent these problems have been addressed by DRE systems.Voter intention in a DRE system is a binary input from a digital device so ambiguity cannot exist. However,it is well known that processingerrors, computer memory corruption or computer memory loss can occur in digital devices and there have been well publicized reports suggestingthat this may be the casewith someprior art DRE systems. If this type of failurewere to occur,voter intention is permanently corrupted or lost and a new election would have to be held. In addition to the aforesaid disadvantages,the prior art solutions generally do not enable an election official to effectivelyaudit voting results and/or verify the integrity of

CROSS REFERENCES This applicationis a continuationof U.S. patent application Ser. No. 11/121,997, filed May 5, 2005, and entitled "System, Method and Computer Program for Vote Tabulation with an ElectronicAudit Trail";which claimspriorityto Canadian Patent Application No. 2,466,466, filed May 5, 2004, and entitled"System,Methodand ComputerProgram for Vote Tabulation with an Electronic Audit Trail"; the entire disclosures of which are incorporatedherein by reference.

10

15

FIELD OF INVENTION This inventionrelates generallyto technologiesfor tabulating votes. This invention relates more particularly to ballot boxes with vote tabulationcapabilities.The invention also relates to method for tabulating votes in an election. BACKGROUNDOF THE INVENTION Numerous voting technologies are known. These technologies are generally directed to permitting votes to be recorded with efficacywhile maintainingthe secrecy of the ballot. One categoryof votingtechnologyinvolvesthe use of the paper ballot. Variousmethods exist for collecting and tabulating votes recorded by paper ballots. Prior art has been created to rely on paper ballots while using modern digital technology. For example, "Optical-mark Based Systems" allow voters to mark paper ballots and feed the ballots through an automated optical ballot scanningtabulator into sealed ballot boxes. The optical scanningtechnology determines the nature of the vote and the computer technology provides a means to quicklytabulate results. The results are transmitted to a central computer by a variety of transmission technologies.All results are tallied at the central computer. Another category of voting technology is best described as "MechanicalVoting Systems". These systems generally Utilize mechanical counters for counting votes. These systems include lever machines and punch card ballots. Another category of voting technology is the "Direct Recording Electronic" (DRE) system. This type of system permits voters to enter their vote into a digital system by means of an electronic interface such as a touch screen, mouse or scrolling cursor. Generally speaking, an interface devicein eachvoting stationis linkedto a Computersystem. The results compiledby each set of voting stations are then transmittedto a central computer system using a variety of computer architecturesand transmissionmodes. In a variation of the DRE system,the entry of a vote into a digital system is linked to a paper receipt that is used as a voter verifiedpaper receipt. This paper receipt can be used in a paper-based audit procedure-an operation that is impossible in a true DRE system. There is a need for voting technologiesthat provide the ability to audit the election results. Democracydepends on accurate and incorruptibletabulation of voter intent; so by extension,voting technologiesmust be able to demonstrate vote tabulation integrity.

20

25

30

35

40

45

50

55

60

65

US 9,870,666 B2 3

4

the vote. For example, the above mentioned prior art technologies do not permit an electionofficialto pick a ballot at random and examine the results of the optical scan algorithms that determinedvoter intent. This ability to compare the paper ballot with the results collected by a vote system allows the election official to verify the accuracy of each votingunit and ensurethat there are no randomor systematic tabulation errors in the voting system or subsystems. There is a need for a voting system and ballot collection apparatusthat enablesobjectivemeasurementof voter intention and a method to verify accurate data collection to demonstratethe integrity of the vote.

cartridges from a plurality of ballot boxes is retrieved and stored to a central memory linked to the central processing computer. One aspect of the present inventionis that the association of the vote stamp with the corresponding digital image enables rapid and cost effectiveverificationof the integrity of the processes. For example, a random selection of a definedpercentage of ballots processed in accordancewith the invention can be examined for accurate collection of voter intent by calling up from the central memory the digital images and correspondingvote stamps. (for example from ballots received in a particularpolling station) and by comparingthe digital image displayedon a computerscreen with the results of the processing thereof as shown by the data included in the vote stamp. Another aspect of the present inventionis that it provides the objectivity and speed of a digital process in connection with the use of paper ballots in an electoral process. A writtenrecordof the electoralchoicein the form of the paper ballot is maintained as well as a cost savings as fully electronicvoting (for example using touch screens) is generally more costly. One object of the present inventionis to provide a system and method that enables an electoral process that can be audited efficiently,accurately and transparently. Another object of the present invention is to include the aforesaid auditing capabilities in a machine that can be easily accessed by any persons with visual, motor or language disabilities.

10

SUMMARYOF INVENTION 15

A tamper-proofand secure ballot box includes a slot that permits paper ballots (which include one or more voter selection entry areas to be manually completed by a voter using a suitable marking device) to be fed into the box by means of a paper feed mechanism. 20 The interior of the box includes a ballot processing device. The ballot processing device includes a computer and a digital scarmingdevice. The ballot processing application of the present invention is loaded on the computer. The digital scanningdevice scans each paper ballot fed into 25 the box to create a digital image thereof, in a manner that is known. The ballot processing application analyses each digital image based on predeterminedcriteria. The predetermined analysis criteria include co-ordinatesof marking locations, 30 co-ordinates of ballot identifiers, locations of relevant scripts, and pixel-based threshold values for determining voter intent (marked, unmarked, ambiguous). The ballot box includes an alarm device that signals incorrect ballots or when a ballot has been determined to 35 require resolution of intent by the voter (erroneous or ambiguouslymarked voter selection entires). In this case, the ballot is returned by the feeding mechanism, following which normal procedures are followed to cancel the ballot and issue a new ballot to the voter. 40 Another aspect of the inventionis that the ballot processing applicationcreates a "vote stamp" based on analysis of each digital image. This vote stamp is attached to a digital image of the scanned ballot and stored as a single data file to a memory linked to the computer.The vote stamp may 45 includea randomlyassignedunique identifiercorresponding to a particular digital image of a ballot. The vote stamp also may includedata correspondingto the results of the analysis of the digital image conducted by the ballot processing application.Another aspect of the vote stamp is that it may 50 includeother datarelevantto the electionprocess such as the polling station where the particular ballot was scanned. The paper ballot is retained within the ballot box of the present invention in a secure marmer and then handled in accordancewith the specificrules of the election in which 55 the ballot box is used. The various data files correspondingto the digital ballot images and corresponding vote stamp are stored in the memory in random order.The voting selectionof a particular individualcannot be discernedby individualsobserving 60 the order in which ballots are fed into the ballot box. The computer also includes a suitable back-up storage device and related utilities to reduce the risk of data loss from the memory. The systemof the inventionincludes a centralprocessing 65 computer.The memory linked to the computerconsistsof a computer cartridge. Data from a plurality of computer

BRIEF DESCRIPTIONOF THE DRAWINGS A detailed description of several embodiments of the present invention is provided herein below by way of example only and with referenceto the followingdrawings, in which: FIG. 1 is a flowchartillustratingthe methodof the present invention,in accordancewith one particular aspect thereof. FIG. 2 is an illustration of the tabulation unit of the present invention, in a cross-sectionthereof. FIG. 3 is an illustrationof a paper ballot that provides an example of a paper ballot processed in accordancewith the present invention. FIG. 4 illustrates a scanned and processed image of the paperballot illustratedin FIG. 3 that includesthe vote stamp footer of the present invention. FIG. 5 is a diagram illustrating a system architecture representative of one embodiment of the system of the present invention. FIG. 6 is a further scannedimage of anotherpaper ballot that includes a vote stamp footer, wherein the vote stamp footer reflectsprocessingresults inconsistentwith the voter intention. In the drawings,preferred embodimentsof the invention are illustrated by way of example. It is to be expressly understoodthat the descriptionand drawingsare only for the purpose of illustration and as an aid to understanding,and are not intendedas a definitionof the limits of the invention. DETAILEDDESCRIPTIONOF THE INVENTION One aspect of the present invention IS a method that includes the following steps: 1. Establishingone or more criteria for analyzinga plurality of paper ballots, the paper ballots including one or more voter selection areas, which criteria include threshold

US 9,870,666 B2 5 values for determining whether a particular voter selection area has been filled in, has not been filled in, or it is ambiguouswhether or not it has been filled in; 2. Scanningat least one of the paperballotsusing a scanning device, the scanning device generating a digital image document of the each paper ballot; 3. Analyzing the digital image document to determine whethera particularvoter selectionareahas been filledin, has not been filledin, or is ambiguousas to whetheror not it has been filled in, so as to definea series of processing 10 results for each digital image; 4. Creation of a vote stamp which includes the processing results and attaching the vote stamp to the digital image document; and 5. Comparingthe processing results with the digital image 15 document corresponding to a paper ballot, thereby enablingthe electionresults to be audited on a per ballot basis. A further aspect of the method of the present inventionis that the threshold values are based on pixel count. The 20 threshold values are established by election officials, as particularizedbelow: In anotheraspectof the invention,the vote stampincludes other data relevantto the electionprocess such as the polling station where the particular ballot was scanned. 25 Furtheraspectsof the method of the present inventionare explainedbelow. Overall, the method of the present invention is illustrated in FIG. 1. FIG. 2 illustratesone particularembodimentof the invention which consists of a hybrid paper/electronicvote tabu- 30 lator 10. The hybrid paper/electronicvote tabulator or tabulation unit 10 includes a lock down type ballot box 12, earphones 14, and a pendant manual trigger 16. As voters come into the polling location, they are generally processedby the election officialwho determinesvoter 35 eligibility (based on local election rules), and also determines the proper ballot for the voter. Based on the voter's requirements;they are either issued a paper ballot, or issued an 'audio' ballot, in accordance with one aspect of the 40 present invention. FIG. 3 illustrates a representativepaper ballot 17, which includes a plurality of voter selection areas 18. In the case of the particular paper ballot 17 illustrated,there is also an election officialarea 20, which appears in most ballots, and is used by the election official to validate the particular 45 ballot. Usually the validation takes the form of a signature in the election official area 20 by the election official, however, other means of authentication can also be used. The election officialarea 20 servesto mark the paper ballot 17 as having been officiallyissued. 50 It shouldbe understoodthat the paperballots 17 can either consist of pre-printed ballots or a ballots printed locally on-demandby means of a regular laser printer. The voter will generally take the validated blank paper ballot 17 to a private area (such as a desk with privacy 55 barriers), and make a mark or marks (this can be done in numerousways such as fillingin a box, making an 'X', etc) beside the chosen candidate(s)with a permanent marker. For the sake of clarity,this disclosuregenerallyrefers to selectionof a candidateor candidates,however,some votes 60 relate to other questions such as referendum questions and the like. The present inventionextendsto votes obtainedfor a voter mandateon all matterswhethercandidates,weighted candidate votes, referendum questions and the like. The voter then generallytakes the markedpaper ballot 17 65 to the tabulationunit 10, where it is fed through the hybrid tabulator 10, as shown in FIG. 2. Secrecy of voter intent is

6 preferablymaintainedby using a confidentialityfoldermade from paper (not shown). The tabulation unit 10 includes a ballot processingdevice22. The ballot processingdevice 22 includes a computer (not shown) and a ballot reader or digital scanningdevice 24. The ballot processingdevice 22 also includesa known paper feed mechanism26. The ballot 17 is fed through the paper feed mechanism 26 and the digital scanning device 24 takes a high resolution optical scan of the ballot 17. In one particular aspect of the invention, the tabulation unit 10 takes the resulting high resolutionscannedimage (not shown)of the entireballot 17, and saves one copy of this image to non-volatile memory (not shown) linked to the computer (not shown) so as to create a permanent record thereof. Loaded on the computeris the ballot processingapplication 28 which constitutes the computer program aspect of the present invention.The ballot processing application 28 randomly assigns a filename to each scanned image (not shown)to ensurethat the order in which the ballots 17 were scanned remains private. The ballot processing application 28 also sendsanother copy of the images (not shown)to the computer (not shown) for image recognition. In a particular aspect of the invention,the computer (not shown) is linked to the digital scanning device 24, and the ballot processing application 26 defines an image recognition routine (not shown) that is applied to the digital scanning device 24. One aspect of the image recognitionroutine (not shown) is that the ballot processing application 28 enables the computerto selectivelyrecognize specificareas of each image (not shown) and analyze such specificareas as particularized below to define a series of processing results (not shown) associatedwith the particularballot 17. For example,the ballot processingapplication28 enables the computerto analyzethe various securitymarkingson the ballot 17 to ensure it is a valid ballot, perform a pixel count checkto verify that the electionofficialarea 20 was initialed by the election official to ensure the ballot was properly issuedto an eligiblevoter, and perform a pixel count of each voter selection area 18 on the ballot 17. In addition, as further examples of such processing results (not shown), dependingon the pixel count of each markingbox, the mark is eitherclassifiedas a 'vote', a 'non-vote', or an 'ambiguous marking'. These classificationsare based on the total pixel counts of the marking areas ("marking area" or "marking areas" refer for example to voter selection areas 18 or election official areas 20 and other similar areas) of the ballot, and the corresponding pixel levels defined by the election officialsat a given time prior to the election. The electionofficialsclearlydefine,in pixels,the minimumpixel count that is to be classifiedas a 'vote', the maximumpixel count (if any) that is to be defined as a definite 'non-vote', and a range of pixels in between those values that will constitute an 'ambiguous marking'. These pixel values are loaded on each tabulation unit 10. After calculating these pixel values for each marking area, the tabulation unit 10 will return the ballot (by operationof the paper feed mechanism 26) to the voter with an appropriateerror message for any of the followingscenarios(as a particularillustrationof the present invention): any marking area was classifiedas an 'ambigious marking'; too manymarkingareasin one categoryWereclassifiedas 'votes' resulting in an "over-vote"; no marking areas in one or more categorywere classified as 'non-votes', resulting in a blank ballot; and/or

US 9,870,666 B2 7

8

no pixel count was recorded in the election officialarea 20, indicating that the ballot was not initialed by an election worker. It shouldbe understoodthat the tabulationunit 10 can be programmedto allow a voter to verify its ballot in the case of over-voted or blank ballots, thus preserving the voter's right to cast an over-votedor blank ballot. If the ballot was not returned to the voter for any of the abovereasons,the computer(not shown)determinesthat all of the votes pertainingto the particularballot 17 have been cast by that ballot, and appropriately increments a main counter (not shown) that forms part of the memory (not shown),in a particularembodimentof the present invention. In addition, once the determinationof the total votes for that ballot 17 has been made, by further operation of the ballot processing application28, the computer (not shown) appends a footer 30 to the saved ballot image (not shown), which contains all of the processingresults (not shown) for that specificballot 17, as shownin FIG. 4. The footer 30 can also be understoodas a "fingerprint"or a "vote stamp" that is attachedto each image (not shown).The presentinvention thereby permits the creation of one random, clear, image of every ballot 17 along with all the informationthe tabulation unit 10 has compiledfor that particularballot 17 that can be later viewed by any software image viewer (not shown). Upon successful completion of the ballot processing, in accordance with the present invention, the ballot 17 is passed the rest of the way through the tabulation unit 10, where it drops into the ballot box 12, where it mixes freely with the otherballotsto preservethe anonymousorderingof the ballots (as shown in FIG. 2). In a particular aspect of the present invention, means is provided for voting by an individualwith a challenge such as visual impairment, illiteracy, inability to read the language in which the ballot is written, physically impairment with only limited motor abilities, etc. In such case, in accordancewith another aspect of the method of the present invention, the election official initializes an 'audio ballot' (not shown)which is administereddirectlyby the tabulation unit 10 (in one particularembodimentthereof).By operation of another aspect of the ballot processingapplication28 of the present invention, the election official initializes, or activates, the audio ballot by using an electronic key (or some other suitable mechanism), and ushers the voter to a voting area wherethe voter can put on the ear phones 14 and take hold of the handheld pendant that includes the manual trigger selector 16. The audio ballot (not shown) is best understood as an audio version of the ballot 17, which is stored to the memory (not shown) associated with the computer (not shown). The voter negotiates through the ballot 17 through the audio interface and makes candidate selections via the manual trigger 16. Every time the voter makes a selection, and everytime a voter completesvoting for every individual race, the computer(not shown)asks the voter to confirmthe choices after it is audiblyreviewed.Upon completionof the audio ballot, a final review is read back to the voter, and the voter is askedone more time to confirmthe selectedchoices. This audio clip of this review, along with the subsequent sound of the voter confirmingthe review, is recorded and randomly stored in memory (not shown)by operationof the ballot processingapplication28. In addition,once the voter is finishedwith this final review, the election officialgives the voter one last chance to cancel the ballot and start over. Once this option has been denied by the voter, the election officialuses their electronickey to cast the audio ballot. By doing so,the votes fromthat audioballot are appendedto the

ongoing tally of the tabulation unit 10 (as particularized above), the sound bites are saved to the memory, and optionally a printer linked to the computer (not shown) prints the votes summaryof the audiovote onto a paper chit. The chit then falls into the ballot box 12 in orderto maintain secrecyor is printedin encryptedformfor audioplaybackon either the tabulationunit 10 or external device. This aspect of the method of the present invention is also illustrated in FIG. 1. At the end of voting, the tabulation unit 10 includes an ongoing tally for all the ballots that passed through that particular machine, and those results are transmitted (by various known methods and technologies) to a central location which includes a central election database 32, as illustrated in FIG. 5. The various election results from the varioustabulationunits 10 are summedtogetherin a mauner that is known and storedto the central election database32, in a marmerthat is known. It should be understood that one aspect of the present inventionis a systemconsistingof the tabulationunit 10 of the present invention,and which includesthe computer(not shown) and the ballot processing application 28. Another aspect of the present inventionis a further systemconsisting of a plurality of tabulation units 10 linked to a central computer (not shown), the central computer (not shown) being linked to the central election database 32. The systemand computerprogram aspects of the present inventioncan be providedin a mannerthat is knownto those skilled in the art. Particularhardware and software components are describedbelow for illustrationpurposes. TechnicalDescriptionof the TabulatorHardware The tabulation unit 10 is generally a self-containedunit made up of severaldifferentsubcomponents.The following is a list and description of representative subcomponents. Motherboard In one particularaspectof the presentinvention,the ballot processingmechanism 22 includes a motherboardthat provides the functional block of the tabulation unit 10, and which controls electronic processing of the tabulator process. The motherboard (not shown) preferably includes severalinterfaces,includingan interfacewith the paper feed mechanism 26, an audio/general purpose input output (GPIO) interface, a scanning card and image heads (associatedwith the digitalscanningdevice24),the power supply board, a thermal printer,and the user interface34 illustrated in FIG. 2. The motherboard(not shown)includesthe computer(not shown) consisting of a known microprocessor.The motherboard (not shown) along with its processor (not shown) can be made to operate with a variety of softwareoperating systems, but currently is utilizing either WINDOWST™ CE.Net or uClinux™. The processor (not shown) may consist of an x86 INTEL™ based processor (for the Windows CE.Net version) or, for example, a MOTOROLA COLDFIRE™processor (for the uClinux™ version). The motherboard (not shown) has various sources of memory including, for example, onboard flash memory where the operating system kernel resides, as well as various hard drive options including an on-boardATAflash drive and a removable compact flash drive. Paper Feed Mechanism The paper feedmechanism26 is that part of the tabulation unit 10 that physically controls the movement of the paper ballot throughthe tabulationunit 10. In a particularembodiment of the present invention,the paper feed mechanism26 is of a very similar in construction to todays modem sheet-fed scanning systems (or sheet-fed fax machines) in

10

15

20

25

30

35

40

45

50

55

60

65

US 9,870,666 B2 9

10

that it incorporatesa systemof rubberizedwheels connected to a stepper motor that are responsible for advancing the paper either forward or backward through the paper feed mechanism. The paper feed mechanism 26 also houses a object sensor system which is a series of LED/photodetectors used to detect a sheet of paper that has been inserted into the opening slot of the paper feed mechanism 26. Also included on the paper feed mechanism 26 is a double paper sensor, which is a commercially available module that is capable of detecting more than one sheet of paper. Lastly,the paper feed mechanism26 also houses two separatelypackaged contact image sensor (CIS) units (one above the paper feed plane, and one below allowing for duplex imaging), which are commerciallyavailable and in large scale use in commercial fax and scanning machines. The paper feed mechanism26 is preferablydesignedin such a way as to allow the rubber wheels to advance and reverse the paper through the paper feed mechanismand across the mounted CIS units in such a manner as to allow no more than 7 degrees of paper skew, and in such a manner as to ensure that the paper does not bind with any part of the tabulator, thus ensuring that the paper does not jam inside the tabulator. The opening slot 36 of the paper feed mechanism 26 is extendedout through a front slot 38 of the ballot box 12 for the insertion of the paper, and a rear slot is 40 extended throughthe rear wall of the paper feed mechanismensuring that the paper is ableto freely exit the paper feed mechanism and fall directly into the ballot box 12. Scanning Card and Image Heads The digital scanning device 24 of the present invention preferably includes a series of imaging heads or modules (not shown), which can be either be of the contact image sensor (CIS) or charge coupled device (CCD) type. When prompted, each imagingmodule (either CIS or CCD) takes a one dimensionalimage of the ballot paper that is in direct contact with the module, producing a series of black and white or grey-scalepixels that represent the image for that particular line of the paper. By couplingthis process with a single line advance of the steppermotor (i.e. advancingthe paper by one line) and repeating, a very high quality two dimensional image of the entire sheet of paper is created. The imaging modules interface directly with the image controller.In one form of the tabulation unit 10, this controller is in the form of a separatelymountedcustomprinted circuit board (PCB). This board contains an application specific integrated circuit (ASIC) chip that has been designedfor specificuse in scanningapplications.This chip is commerciallyavailable and is in wide use in commercial scanningproducts.In its other form, this imaging controller resides directlyon the Coldfireprocessorwhich is locatedon the main motherboard.In this version of the controller,the softwarethat is encoded on the ASIC chip is ported directly onto the main processor, thus eliminating the need for a separate PCB. In either case, the imaging controller is one particular aspect of the ballot processing application 28 of the present invention. The function of this imaging controller is to control the feeding and scanningof each ballot 17. When 'armed' (i.e. the main tabulator applicationsendsthe signal to the image controllerto be ready to accept and scan ballots) the image controller monitors the paper sensor hardware. When a ballot 17 is insertedinto the openingof the tabulator (i.e. the front of the paper feed mechanism system 26), the paper sense hardware trips, and sends the signal to the image controllerto start the scan routine. In this scan routine, the image controllerrepeatedlytriggersthe imagingmodulesto

scan a line and then advancesthe paper feedingwheels (i.e. via the stepper motor) and repeats until the paper has been completelyscanned.Whenthe paper is completelyscanned, two complete images are sent by the imaging controller to the main tabulator application for processing. During this time, it is important to note that the paper has not been completely fed through the paper feed mechanism. Even though the ballot has completelypassed over each imaging head, the bottompiece of the ballot is still being held by the rear rubberizedwheels. It is at this point where the imaging controller awaits instructions from the main processor or computer (not shown) for its next instruction. This next instructioncanbe eitherto completelyreturnthe ballotto the voter (i.e. reversing the direction of the stepper motor allowing the rubberized wheels to move the paper back through the machine and out the front slot) or to accept the ballot (i.e. forwardingthe stepper motor a predefinednumber of steps to allow the ballot to continuethrough past the rear slot 40 of the tabulator, allowing the ballot to fall into the ballot box). After this instruction, the controller will 're-arm' itself in preparation for the next paper ballot until suchtime as the main tabulatorapplicationtells it to turn off. Power Supply The power supply is generally a separate PCB that convertsthe incomingpower voltage (either 120VAC,240 VAC, or 30 VDC) to a DC voltage that is required by the other PCB's in the tabulator.Specifically,-5VDC, +5VDC, +12VDCand -12VDC are the outputs of this PCB, and are used to power the motherboard, the scanning board, the user-interfacemodule, and the thermal printer. Audio/GPIO Card The audio/GPIO card is either in the form of a separate PCB, or built-in on the motherboard. Its function is to provide an audio output and a GPIO input to the motherboard. The card interfaces to two mini 3.5 mm connectors that are mounted on the rear plate of the tabulator; one is where the earphones 14 plug in, and the other is where the handheldpendantthat housesthe manualtrigger 16 plugs in. This card allowsthe main applicationto play various sound files via the earphone jack, and accept user input via the handheld pendant. Thermal Printer This is a commercially available thermal printer which interfaces directly with the motherboard, in one particular embodimentof the present inventionmentionedabove. The printer is mounted in such a way that the paper output is aligned with a slot in the tabulator cover, allowing the printed piece of paper to feed throughthe tabulator cover to the outside. When sent a print job by the motherboard,the printer produces a printout on a thermal paper roll. User Interface The user interface34 is a knowninterfacedevicemounted to the tabulator cover, which possessesboth an information display screen and a method of user input. In a preferred embodimentof the presentinvention,there are currentlytwo different types of this module. The first is a TFT display screencoupledto a resistivetouch-screeninput device.Both are commerciallyavailable pieces of hardware. The second type of user-interface module available is a module that possesses an LCD display screen with several keypad buttons. Both modules interfacedirectlywith the motherboard, and are controlledby the ballot processingapplication28. It is through this device that election officials are able to interface with the tabulator. This interface is used for a numberof situations,includingenteringadministrativepasswords, settingup audioballots,finalcastingof audioballots, printing various types of election reports, accepting or

10

15

20

25

30

35

40

45

50

55

60

65

US 9,870,666 B2 11

12

returning over-votedor blank ballots, etc., which constitute further functions of the ballot processing application 28. Chassis and Exterior Package The chassis is made up of a metal base and metal rear panel which is mountedto the base at 90 degrees.The power supply PCB, the scauner PCB, the motherboardPCB, thermal printer and audio/GPIOboard are all mounted to this metal chassis.The paper feed mechanismis then mountedto the base as well,but sits abovethe above othermodules.The user interface module is fastened to the paper feed mechanism cover (which can be either plastic or metal) which in turn fastens to the metal base and rear plates via tamperproof security screws (i.e. torx screws). Descriptionof Scauning Software As mentioned above, one aspect of the ballot processing application 28 of the present invention is an application layer that calls the scanningfunctionthat points to a matrix. When a ballot 17 is scanneda driver of the ballot processing application28 is responsibleto copy the digital scan of the document to a specified location on the memory (not shown). Whenthe scanneris enabledit is in ready/activemodeand will await the insertionof a ballot 17 in the scanner.In this mode of operation,the digital scanningdevice 24 will await the insertion of a ballot to begin the scan. The operationof the scanner and driver during the scanning process is outlined below: 1. When a ballot 17 is inserted into the digital scanning device24, the front ObjectSensorwill senda signalto the scanningboard which will notify the driver. If the driver is in active mode, it will start scanningthe document. 2. At the start of each line scan, the driver will send a 'start pulse signal' to the CIS module (where a CIS module is used, as explained above). This prompts the CIS module to place an analog output voltage on its Vout pin that is representative of the darkness of the first pixel. This is followedby a sequentialseries of Voutvoltages for each of the pixels in the line. These signals are spaced by the internal CIS clock, which is set by the driver. As the digital scanningdevice 24 is going through its line scan, the scanningboard will convert each voltage signal to a digital signal, in a manner that is known. The threshold value of this quantizationis configurable. 3. The resultantdigitizedimage (not shown),again, is stored in the predefined location of the memory (not shown). 4. When the back Object Sensor is triggered, the scanning board will notify the driver.At this point the driver will notify the application layer (via a function call) that the document has been scanned, imaged, and is sitting in matrix form in memory. The driver will also, via this function call, let the application know the size of the matrix. Here, the driver will be put in a wait state. It will wait for instructions from the application layer. 5. A driver definedAPI is provided so that the application can then instruct the driver to either pass the document completelythrough (forward)the digital scanningdevice 24, or completely reverse the direction of the stepper motor, returning the entire document back through the front of the unit. Also if the driver detects any errors in the document (i.e. double sheet, paper jam, etc.) then it will reverse the document and notify the application layer of the error. During this disabled state the driver will disregard the beginningof document signal from the device and not scan any documents (i.e ballots). The digital scanningdevice 24 is initially in this inactive state. It will be in this state until

the initializationroutine is completeand everythingis okay. It can be put back in this inactive state by the application layer. The overall speed of a scan from once the paper is sensed to when the driver notifiesthe applicationlayer that a ballot 17has been scannedis generallyunder one (1) second,using existingtechnology,which is suitablefor the purpose of the present invention. The driver describedwill generallyallow the scan of any sized document up to a maximum of 21 inches long. Descriptionof RandomAuditing Process As described above, once the vote has been completed, the vote result files are transferred to the central election database 32. Election officialscan now perform a random auditing process, in accordancewith another aspect of the invention,to ensure that the tabulation units 10 have functioned correctly (and also to verify the integrity of the vote overall).This can be in the form of a spot check,a structured check to gain a certain confidence rate in the tabulator integrity,or a complete check of every ballot cast. For the sake of clarity,oncethe vote result filesfrom each tabulation unit 10 have been transferred to the central computer (not shown),there will be one multimediafile for every ballot cast during the election, complete with the corresponding processing results (not shown) or vote stamped 'fingerprint' information. In the case of paper ballots, there is one image file for every ballot cast, and in the case of audio ballots, there will be one sound file for every audio ballot cast. Dependingon the auditing strategy used, the auditing committee (for example) can have any number of terminals accessing the ballot files from the central election database 32, as illustrated in FIG. 5. Each auditor working on a separate computer can open as many ballot files as needed. Audit Process The process that allows auditors to verify that any given ballot was tabulated properly is as follows. As explained above, a completeimage of each ballot 17 scannedis saved to the ballot file. In addition, the fingerprint of that ballot (which contains all the pertinent informationthat the tabulator obtained from that ballot, including pixel values for each marking area as well as which candidateswhere voted for) is vote stampedbelow that image by including same in the footer 30. By simply opening any given ballot file on their computer terminal, the election auditor is able to compare what the voter markings to the tabulator interpretation.In the case of the audioballot,the electionauditorcan compare the sound file that the voter reviewed and confirmedwith what the tabulatorrecordedfor that ballot. Thus by simplyopeningone of these multimediafiles,the auditor can verify whether the tabulation unit 10 made an error in tabulating that particular vote. In FIG. 6, we see an example of a ballot that was incorrectly tabulated. It is clear that the voter intended to vote for the write-in candidate 'Nick Salvatore', but the tabulatorregistereda vote for Pierre E. Trudeau.In essence, to verify that one ballot has been tabulated correctly takes only a few seconds. Types of Auditing Strategies The main advantage of this type of fingerprinting and corresponding audit trail is that it easily enables virtually any auditing strategy.On the one extreme,perhapsonly one or two ballot files from each tabulation unit 10 would be checked to verify that the tabulator fingerprintmatched the originalballot information.This wouldprovide somedegree of a sanity check against a systematicproblem,but depend-

10

15

20

25

30

35

40

45

50

55

60

65

US 9,870,666 B2 13 ing on the number of ballots cast at each tabulator, would generatea fairly low confidencerate on the auditingprocess. On the other extreme, would be a 100% confidencerate on the audit. For this, every single ballot file would be openedto confirmthat the tabulator fingerprintmatched the original ballot information. However, depending on how many ballots had been cast during the election and how many audit workerswere availableto carry out the auditing, this process could be too slow. It should be understoodthat the present inventioncan be 10 used in many voting scenarios,includingfor example shareholder votes and the like. Specificaspects of the hardware and software described can be modified without departing from the system and computer program aspects of the present invention. 15 What is claimed is: 1. A method for determining votes recorded on a voter marked paper ballot, comprising: receiving, at a ballot processing computer,optical image data comprising an optical image of a voter-marked 20 paper ballot, the optical image of the voter-marked paper ballot including identifiedvoter selection areas; performing a pixel count of at least one voter selection area, the pixel count identifyinga number of pixels in the optical image that contain a voter marking; 25 determining,at the ballotprocessingcomputer,that one or more voter selection areas of the optical image of the voter-markedpaper ballot have been selectedwhen the pixel count of one or more of the identified voter selectionareas in the optical image of the voter-marked 30 paper ballot meet or exceed a predefinedfirstthreshold value for determininga specificvoter selectionareahas been selected; determining,at the ballotprocessingcomputer,that one or more voter selection areas of the optical image of the 35 voter-markedpaper ballot have not been selectedwhen the pixel count of one or more of the identifiedvoter selectionareas in the optical image of the voter-marked paper ballot fall at or below a predefined second threshold value for determininga specificvoter selec- 40 tion area has not been selected; determining, at the ballot processing computer based at least in part on the pixel count, that the optical image of the voter-markedpaper ballot includes at least one ambiguousmark when the pixel count of one or more 45 of the identified voter selection areas in the optical image of the voter-markedpaperballot fall betweenthe predefined first threshold value and second threshold value; and outputting,from the ballot processing computer,an indi- 50 cation that one or more of the identifiedvoter selection areas are ambiguous when it is determined that the opticalimage of the voter-markedpaper ballot includes at least one ambiguousmark. 2. The method of claim 1, wherein the outputting an 55 indication comprises: identifyingmarks on the voter-markedpaper ballot determined to be ambiguousmarks. 3. The method of claim 2, further comprising: routing the voter-markedpaper ballot to a first location 60 when it is determined that the voter-marked paper ballot includes at least one ambiguousmark; and routingthe voter-markedpaperballot to a secondlocation when it is not determinedthat the voter-markedpaper ballot includes at least one ambiguousmark. 65 4. The method of claim 3, wherein the second location comprises a ballot box.

14 5. The method of claim 3, wherein routing the votermarked paper ballot to a first location comprises: returning the voter-markedpaper ballot to the voter. 6. The method of claim 1, further comprising: determiningone or morevotes based on one or morevoter selectionareas that have been determinedto have been selected. 7. The method of claim 6, further comprising: identifyingan overvotewhen votes for an identifiedrace exceed an allowed number of votes for the identified race. S. The method of claim 1, wherein the predefined first thresholdvalue comprisesa minimumpixel count that is to be classifiedas a vote. 9. The method of claim 1, wherein the predefinedsecond thresholdvalue comprisesa maximumpixel count that is to be classifiedas a non-vote. 10. The methodof claim 1, whereinthe predefinedsecond thresholdvalue that is to be classifiedas a non-vote is zero pixels. 11. The method of claim 1, further comprising: determiningone or more selected votes based on one or more voter selectionareasthat have been determinedto have been selected; and generating a vote stamp comprising the one or more selected votes. 12. The method of claim 11, further comprising: appendingan image of the vote stampto the opticalimage of the voter-markedpaper ballot. 13. A system for determiningvotes recorded on a votermarked paper ballot, comprising: a processor; and a memory in electronic communicationwith the processor,whereinthe memory stores executableinstructions that when executedby the processor cause the processor to perform operations comprising: receiving optical image data comprising an optical image of a voter-marked paper ballot, the optical image of the voter-marked paper ballot including identifiedvoter selection areas, performinga pixel count of at least one voter selection area, the pixel count identifyinga number of pixels in the optical image that contain a voter marking; determiningthat one or more voter selection areas of the voter-marked paper ballot have been selected when the pixel count of one or more of the identified voter selection areas in the optical image of the voter-marked paper ballot meet or exceed a predefined first threshold value for determining a specific voter selection area has been selected; determiningthat one or more voter selection areas of the voter-markedpaper ballot have not been selected when the pixel count of one or more of the identified voter selection areas in the optical image of the voter-marked paper ballot fall at or below a predefined second threshold value for determining a specific voter selection area has not been selected; and determiningthat the voter-markedpaperballot includes at least one ambiguousmark when the pixel count of one or more of the identifiedvoter selectionareas in the optical image of the voter-markedpaper ballot fall between the predefinedfirst thresholdvalue and second threshold value. 14. The system of claim 13, wherein the memory stores further executable instructions that when executed by the processor cause the processor to output an indication that

US 9,870,666 B2 15 one or more voter selectionareas are ambiguouswhen it is determined that the voter-marked paper ballot includes at least one ambiguousmark. 15. The system of claim 13, wherein the memory stores further executable instructions that when executed by the processor cause the processorto perform further operations comprising: determiningone or morevotes based on one or morevoter selectionareas that have been determinedto have been selected. 16. The system of claim 13, wherein the predefinedfirst thresholdvalue comprisesa minimum pixel count that is to be classifiedas a vote. 17. The system of claim 13, wherein the predefined second threshold value comprises a maximum pixel count that is to be classifiedas a non-vote. 18. The system of claim 13, wherein the predefined secondthresholdvalue that is to be classifiedas a non-vote is zero pixels.

* * * * *

16

10

15

20

111111

(12)

1111111111111111111111111111111111111111111111111111111111111 US009870667B2

United States Patent

(10)

Poulos et al.

(45)

(54) APPENDING AUDIT MARK IMAGE

Patent No.: US 9,870,667 B2 Date of Patent: *Jan. 16,2018

(58) Field of Classification Search CPC G06Q 30/0207-30/0277; G06Q 20/382; G06Q 20/401; G07C 13/00 USPC 705/12 See application file for complete search history.

(75) Inventors: John Poulos, Toronto (CA); James Hoover, Montreal (CA); Nick Ikonomakis, Toronto (CA); Goran Obradovlc, Toronto (CA)

(56)

References Cited

(73) Assignee: Dominion Voting Systems, Inc., Denver, CO (US) ( *)

Notice:

U.S. PATENT DOCUMENTS 6,250,548 Bl 6/2001 McClure et al. 7,406,480 B2 * 712008 Seibel 200210143610 Al 1012002 Munyer 2003/0052981 Al * 312003 Kakarala

Subject to any disclaimer, the term of this patent is extended or adjusted under 35 U.S.C. 154(b) by 911 days.

(Continued) OTHER PUBLICATIONS

(21) Appl. No.: 13/525,208

Requisition by the Examiner for Canadian Application No. 2,466,466dated Jul. 8, 2015, from the CanadianIntellectualProperty Office.

Jun. 15, 2012

(65)

Prior Publication Data US 2012/0259681 Al

Primary Examiner - Chrystina E Zelaskiewicz (74) Attorney, Agent, or Firm - Holland & Hart LLP

Oct. 11, 2012

Related U.S. Application Data

(57)

Foreign Application Priority Data

May 5, 2004

(CA)

2466466

(51) Int. Cl. G07C 13/00 (2006.01) G06Q 20/38 (2012.01) G06Q 20/40 (2012.01) (52) U.S. Cl. CPC . G07C 13/00 (2013.01); G06Q 20/382 (2013.01); G06Q 20/401 (2013.01)

~

20 Claims, 7 Drawing Sheets

umuumm~

"" m ~

Cityof Anywhere Gen9rn~~::~~~lacIion

9

18

ABSTRACT

A system, method and computer program for tabulating votes and creating an audit trail is provided. A ballot processing device may include a paper feed mechanism, a computer, a ballot processing application loaded on the computer, and a digital scanning device linked to the computer. The ballot processing application may process the digital image to establish a series of processing results defining one or more voting results for the paper ballot, and also an audit trail. The ballot processing application may process the digital image to define the voting results based on criteria established by election officials, including ambiguous mark criteria. The audit trail enables election officials to verify that particular paper ballots have been processed correctly in accordance with these criteria.

(63) Continuation of application No. 13/463,536, filed on May 3, 2012, which is a continuation of application No. 11/121,997, filed on May 5, 2005, now Pat. No. 8,195,505. (30)

G06T 3/4007 348/272

This patent is subject to a terminal disclaimer.

(22) Filed:

G07C 13/00

I""

ea

~

L__ ea

T(lvgte,compietelyfilli"lhebo~(es) Ona:dto~url:hgi~[s).likethl8: •

::

~b~~~~R

ea

_rorC
!Z'ZlJ I)RobertL /

MAYOR SORDON"[]

COUNCILLOR

~~~R

=-(1;.l~

~~

'II'(IIO,ultllglol

Cardclate3

~eao~ tZ2l

3)Jacquas

ea.

4)Siulllleld~

IZ?<'J

5)Jame5

m

6)Ltij

fZ
1)John

ea

8)Henry

aa

9)Jake

I22ZI

100WIlII..,LM

mI IZi2l

11)Wllfrid

I'ZZ2:l

13)Le$letB.

eaa

14)Lollis S.

12)JohnA.

IZZZl 15)PierteE.

""

0 0 0 ERIKSSON0 FRANKLIN 0 HUDSOPl 0 JOLIET 0 KING 0 LAURIER 0 MacDONALD 0 PEARSOPl 0 CARTIER

CHAMPLAI~

COOK

llMiiesJ

MASSEY

21MlII1eI.JusP. l)ClauS& .$ ~

jJ

O'DANIEL VOIliESPIE

! I Ij: ~

~

... ~

~

0

0

D.

1m! !$I

ea ,,4-,J'm 0 0 ea DO ea 0 0 t:iZI !il 0"_"

~~ ea ea

SAINT·U,URENTO TRUDEAU

~~

::

000000

I~ ~17