A Development of Hybrid Drug Information System Using Image Recognition

Abstract

In order to prevent drug abuse or misuse cases and avoid over-prescriptions, it is necessary for medicine taker to be provided with detailed information about the medicine. In this paper, we propose a drug information system and develop an application to provide information through drug image recognition using a smartphone. We designed a contents-based drug image search algorithm using the color, shape and imprint of drug. Our convenient application can provide users with detailed information about drugs and prevent drug misuse.

1. Introduction

Many people have wondered about their prescription drug, leftover drugs in their cabinet or some drugs found in their teenager’s room. According to the 2013 survey by Korea alliance of patients organization (KAPO), 91% of patients who receive a prescription want to get the detailed information for their doses of medicine [1]. Moreover, the survey by Korea Consumer Agency (KCA) showed that 38.7% of consumers experienced side effects of medicine in the 3 years from 2008 to 2010 [2].

While medicine takers may not aware of the details of a drug's brand name and manufacturer, quite often they are interested in basic facts about indications, dosage, interactions, and side effects of the medicine. In order to avoid over-prescriptions, to prevent medicinal misuse and side effects of medicine, it is necessary for medicine takers to be provided with the detailed information about the medicine.

All approved drugs in most countries are required by the Food and Drug Administration to have a unique imprint. The unique imprint of a drug product is determined by the drug manufacturer. The purpose of an imprint is to enable the identification of drug strength, ingredients, and manufacturer by patients, health care providers and law enforcement officials. Therefore, we designed a drug information system using this imprint.

In this paper, we propose and develop an application to provide the information about a drug through automatic image recognition after a user takes a picture of the drug with their smartphone. To provide drug information, we propose a drug image recognition technique using the color, shape and imprint of drug. We designed a smartphone and smart pad based application to provide a convenient drug search service for users.

This paper is organized as follows. Section 2 surveys the related works. We present drug image recognition in Section 3 and we propose our drug information system in Section 4. Experiment results are presented in Section 5. Finally, we conclude this paper and discuss future works in Section 6.

2. Related Works

Several automated drug information systems proposed in [3,4]. The Medicines Line (1300 MEDICINE) [3] is an automatic telephone information system to provide information about drugs to aged and visually impaired persons. The Chemical Named Entity Recognizer (CheNER) [4] is a tool for automated identification of IUPAC chemical mentions. However, these systems used speech recognition for drug or chemical names.

Several web based drug information systems have developed as shown Figure 1. Web based search systems can be performed in two ways: keyword search and content search based drug image. Current systems include: Drug Information [5], Drugs [6], Drug Guide [7], DrugInfo [8] and Naver Health [9]. However, usually, users must manually input the name, category, national drug, codes, imprint, discernment letter, dosage form, shape or color of the drug to search drug information. Especially with keyword search cases, there is a fatal problem that the user must know the drug name. These web based systems are time-consuming and inconvenient for a drug search because the drug search is user-defined and provides limited features.

3. Drug Image Recognition

We assume that drug is a finished dosage form, e.g., a tablet or capsule that contains a drug substance in association with one or more other ingredients. The drug is identified by shape, color and imprint. A drug imprint is defined as any single number, letter, mark, symbol, logo, or combination of numbers and letters, including national drug code and company names. The unique imprint permits the identification of the drug strength, drug product, active ingredients and the manufacturer of the product. The imprint is marked with an identification code by means of engraving, embossing, debossing, or printing with ink. Engraving is imprinted with a code that is cut into the dosage form surface after it has been completed. Embossing is imprinted with a mark raised above the dosage form surface. Debossing is imprinted with a mark below the dosage form surface.

To recognize the drug image, we use the method of comparing the image registered in the database and drug image that the drug user has taken using a smartphone as a pixel unit of the finely divided image. The programmer or user can decide how to split the image to pixels. We split the image in pixels and then compare the drug image per pixel bundle. Our image recognizers are designed differently according to certain kinds of drug shape and picture discernment. We assort drugs according to shape, hoof shape, color discernment, letters and picture discernment.

Table 1. Setting value examples in each imprint case.

Case Example	Setting Values
	If imprint is printing with ink, 100 pixels per image bundle
	If imprint is engraving, 50 pixels per image bundle
	If imprint is debossing or embossing, 10 pixels per image bundle

shows the different settings for the size of pixels per bundle according to various imprint cases. The more blurry the imprint, the more detailed image comparison is needed. However, detailed comparison will cause the serious problem that the search processing time is longer. Thus, we need a compromise between the search precision and processing time. In drug image recognition, we also consider the interference of background.

Figure 2 shows an example that is well taken photo sharpening without background interference. In this case, it is possible to search exactly for the drug information in a short time because only the drug image part is used as a comparison.

Figure 3 shows a severe case: a photo with background interference. If there is interference of background in the drug photo, it affects the image comparison time.

4. Implementation

To provide drug information using drug image recognition automatically, we designed and implemented a drug information system and smartphone application “DrugWellness”. Figure 4 shows the proposed architecture of our drug information system. Our drug information system consists of web server, data server, and smartphone application.

In order to implement the drug information system and hybrid DrugWellness application, we used Open Computer Vision (OpenCV), Node.js and HTML5. We have constructed a database for drug information and image based on a drug information handbook [10]. Figure 5 shows the screenshots of our DrugWellness application based on an Android environment.

Figure 6 shows the snapshot of detailed drug information after a drug image search. The search results screen is configured as a dynamic web page. This page consists of drug name, drug image, efficacy of the drug, posology and method of administration, special warning and precautions for use, possible side effects, how to store and so on.

5. Experiment Results

We evaluate two aspects of our drug information system: (1) drug image search precision, and (2) the total processing time of end-to-end system. For experiments, we constructed drug image database that consists of 10,000 drug images. We have conducted experiments using 25 drugs with an ink print and 25 engraved drugs.

Figure 7 shows the precision of the drug image search according to the size of pixels per comparison bundle. The result shows that the imprint printed with ink has a higher precision than the engraved imprint. Moreover, the precision drops significantly as the size of pixels per comparison bundle grows. If the size of the pixels per comparison bundle is 100 or less, the precision is more than 90%.

Figure 8 shows the total processing time of end-to-end system according to the size of pixels per comparison bundle. The result shows that the imprint printed with ink has higher performance than the engraved imprint. The total processing time was greater as the size of pixels per comparison bundle was reduced. Also, the processing time difference between the imprint printed with ink and the engraved imprint increased as the size of pixels per comparison bundle was reduced.

4. Conclusions

For safety of drug use, it is essential to provide accurate information about the drug to the user. Previous drug information systems have supported a keyword search based on drug name or category. In this paper, we proposed a drug information system to provide the information about a drug through automatic image recognition. In experiment results, more comparison computation time was required for drug imprints that were engraved or debossed than those printed with ink. In future work, we will study the drug image recognition and search algorithm to improve the search speed.