First known case of the modern machine vision

Summary: The modern machine vision, invented by Professor Antti J. Niemi in 1972, is defined and compared with other, popular MV technologies. His patenting actions and patents granted are then recorded, up to the present day. The last Section returns to Niemi’s early development of his basic idea and refers to some industrial applications of the MV system invented.

Niemi’s technology in comparison with other technologies.

Niemi’s technology for machine vision, i.e. for expression of visually observable information in quantitative form comprises, firstly, image forming optical means, by which the image of the primary, physical object field is formed on the surface of the detector, which consists of an integrated matrix or line array of photosensitive, individually responsive semiconductor elements. The discrete, electric signals from these elements are then read periodically and separate, electric output signals obtained corresponding serially to each detector element. These output signals are coded to serial, digital video signals and brought to a programmable, digital real-time computer programmed to extract quantities characterizing the primary, visible objects and to express them in quantitative, digital form. (Fig. 1; right).

Although the above description may sound familiar to the reader, the MV systems of the presented type were unknown up to April 24, 1972, when Niemi filed his patent application. Before that time, some image analysis systems had been developed, which were based on the use of a conventional television camera tube for production of continuous, analog image signals; these were then sampled and, after an analog-to-digital conversion, brought to a computer for programmed, quantitative analysis. (Fig. 1; left).

It turns out, that Niemi’s approach, which is not based on continuous, analog signals at any stage, is more appropriate to image analysis by computer than the other alternative. It is also more straightforward than such later MV systems, which also use the integrated, photoelectric sensor, but convert the signals obtained from it to continuous, analog signals, which are then brought, through a sampler and an analog-to-digital converter, to a computer. The back-and-forth conversions to and from the continuous signal form involved are not needed by Niemi’s system. (Fig. 1; middle).

About the invention and patenting history.

After Niemi had gained, at his early work and studies, a considerable experience in industrial control technologies, he became interested in control of such quantities and phenomena, which were not readily measurable, although they could be observed by the human senses, especially the vision. During some years, his interest matured to an invention on observation and recognition of visible objects and patterns by a set of discrete, photoelectric sensors, and transfer of the discrete responses obtained to a digital computer, for a quantitative evaluation of such visual features, which characterize the scene or its local details.

He then brought his invention to the form of a patent application, which he filed on April 24, 1972 in Finland and subsequently abroad. He received in UK the Patent No 1.430.420 on March 31, 1976, after a search of a large number of patent classes by the UK Patent Authority, for prior art. The Finnish application was studied for ten more years by the domestic Patent Office, but was not approved.

In the United States, the approval of Niemi’s application was delayed by the actions of the official examiner of the US Patent Office, who repeatedly brought up earlier patents, by which he tried to show Niemi’s invention obvious on the basis of prior art. However, at all stages of this correspondence, Niemi was able to show them inapplicable to the case of his invention. After ten years of such correspondence, he submitted the matter to the Board of Appeals of USPTO, and the US Patent No 4.831.641 “Method and apparatus for analyzing a visible object”, was granted on May 16, 1989, and provided later with USPTO’s Certification of Correction of an error made in USPTO

Niemi transferred in 1982 his British patent rights to a Finnish company, which returned them to him after several years, after they had received independent patents for a particular type of classifier for moving particles. Since 1989 he approached also several US companies, which, however, proved passive to his contacts. This was caused by a satisfaction with established TV and analog technologies, which made it difficult to Niemi to convince potential users of the benefits of his approach, but another reason was Mr. Jerome Lemelson’s MV related activity, of which he was not informed before 1999.

Lemelson had filed his first MV related patent applications, which were based on traditional TV technologies and hardwired electronics, in 1950’s and been able to keep up that originality by filing new applications based on the earlier ones, before these had resulted in patents. By such means, he dominated the field of MV systems and had been able to collect, together with the Lemelson Foundation, a considerable property from industrial users of MV and related technologies in USA (See e.g. Fortune, May 14, 2001, pp. 203-216), making all MV related ideas reluctant to US industries.

Lemelson stated in his continuation-in-part application of May 18, 1972 (i.e. 24 days after Niemi’s original filing date) i.a. “a solid state image sensor containing a suitable number of light sensitive elements”. It resulted in the Patent No 4.118.730, “Scanning apparatus and method”, granted on Oct. 3, 1978. – Any detailed comparison of Niemi’s technology with that of the Lemelson’s patent family is impossible in the available space, but by way of an overall statement, Lemelson e.g. did not specify the interface of the stated sensor with his analog or hybrid signal processing system nor constructed an operating device according to his patents.

Such a comparison is not timely either, because Lemelson’s set of MV related patents and applications was finally taken to the court by a group of US industries and declared invalid on Jan. 23, 2004; this was then confirmed by the US Court of Appeals on Sep. 9, 2005. However, by Prof. Niemi, his Patent is still, at this writing, fully valid in USA, dominating alone the general field of machine vision.

Development and applications of the new approach to Machine Vision.

The need of a more practical instrument for image understanding was sensed by Niemi at the time of his visit to the Achema Days held in Düsseldorf, Germany, in June 1969, and to the connected exhibition. At this occasion, an optical company demonstrated a microscope which had been provided with a television camera tube and rather complex electronics for sampling, analog-to-digital conversion and transmission of the image information to a digital computer programmed for evaluation of features of the object inspected.

He wondered, if the same effect could be reached by simpler means, and compared the exhibited system with the human vision and eye, which forms optically a continuous image of the object and then converts it, by means of the rods and cones of the retina, to locally discrete nerve signals, which are processed by the brain. A search of literature, made after the trip, did not reveal any technological prior art case of this type and suggested, therefore, the construction of a corresponding network of e.g. photosensitive transistors or diodes connected, by an appropriate interface, to a computer.

The question of practical realization of such a network made Niemi to contact one of his colleagues, who was an expert in electronics and could inform him, that corresponding networks had recently been brought to the market in the form of an integrated line array or matrix of photodiodes on a silicon chip; it turned out that e.g. matrices of up to 8x8 elements were available commercially. After another search of literature had not revealed a prior case of a technological machine vision system of related structure, Niemi wrote and filed a patent application providing it also with statements on some example cases and processes to which the new system could be successfully applied.

At the same time, Niemi read a description of “Shakey”, i.e. a robotic arm constructed in Stanford Research Institute and used for manipulation of mechanical objects observed within the visual field of a TV camera. Before this, he had already, during several years, followed the uses of the standard industrial robots, which had been provided commercially since 1962 to pre-pro-grammed handling of parts in mechanical industries. In 1975 his laboratory, i.e. the Control Engineering Laboratory of Helsinki University of Technology received an up-to-date industrial robot IRb-5 from ASEA (ABB), for instruction, study and research of industrial automation. This robot included a micro processor as the central control unit and electric servo loops for movements in 5 degrees of freedom. It was provided in the Laboratory with a two-way connection to an existing general-purpose real-time computer and, further on, to Reticon 50x50 matrix camera.

The result of this work was the universally first system, in which an industrial robot of standard construction was controlled, by an external computer, to visually recognize and automatically handle different, moving and randomly located objects, while the earlier, related systems had been individual laboratory constructions which missed the versatility, robustness and other practical properties of the standard industrial robot. – During the following years, the system described was expanded to a pilot scale, machine vision controlled and flexible manufacturing cell with several machine tools. In 1990’s, the basic,MV controlled robot system was moved to the Museum of Technology in Helsinki. Parallel to these developments, Niemi became interested in the arrangement of the illumination and optical imaging of the object field in relation to each other in such a manner, that the characteristic features of the objects are enhanced and unwanted features, such as bright reflections, subdued or avoided. One such object is the dry line, which indicates the disappearance of the even, watery surface of the fibrous pulp at the wet end of the paper machine. For this case Niemi has developed three methods, the most recent of which is based on reflection and scattering of a scanning laser ray by the pulp surface and elimination of the background light produced by the general illumination of the factory hall, by means of optical filters.

Interview by Mauno Kemppainen

Prof. Niemi was born 1928 in Tampere, received his M.Sc. Degree in Helsinki University of Technology in 1954 and his Ph.D, degree in 1966. He has worked in various upper management positions in Finnish industry and also as researcher, teacher and professor in Finnish universities as well as researcher at Purdue University, at University of Florida and at University of Rome. He has worked for several international scientific and engineering societies and has received several international and Finnish awards. Contact Information: Antti J. Niemi, Professor, Ph. D. (eng.) E-mail: antti.niemi@hut.fi