Brief Summary: This randomized phase II trial is studying how well erlotinib hydrochloride works in treating patients with stage I-III colorectal cancer or adenoma. Erlotinib hydrochloride may stop the growth of tumor cells by blocking some of the enzymes needed for cell growth and by blocking blood flow to the tumor. Erlotinib hydrochloride may also stop tumors from growing or coming back

Detailed Description: PRIMARY OBJECTIVES: I. To test the hypothesis that erlotinib (erlotinib hydrochloride) doses as low as 25 mg will decrease aberrant crypt foci (ACF) phosphorylated extracellular signal-regulated kinases (pERK) levels from baseline (pre) to post erlotinib treatment. SECONDARY OBJECTIVES: I. To test the hypothesis that additional epidermal growth factor (EGF) inducible biomarkers will decrease from baseline (pre) to post treatment with erlotinib 25 mg, 50 mg or 100 mg orally (PO) once daily (QD) therapy. II. To determine the mean decrease from baseline of the ACF: normal mucosa pERK ratio pre and post 8-30 days of erlotinib. III. To determine erlotinib concentration in plasma and colorectal tissue at 25 mg, 50 mg and 100 mg doses after 8-30 days of therapy. IV. To determine the incidence of rash, diarrhea and other side effects of low dose erlotinib. OUTLINE: Patients are randomized to 1 of 3 treatment arms. ARM I: Patients receive 100 mg of erlotinib hydrochloride PO and two 25 mg of placebo PO QD. ARM II: Patients receive 50 mg of erlotinib hydrochloride PO and one 100 mg of placebo PO QD. ARM III: Patients receive 25 mg of erlotinib hydrochloride PO and one 100 mg of placebo and one 25 mg of placebo PO QD. In all arms, treatment continues for 8-30 days in the absence of disease progression or unacceptable toxicity. After completion of study treatment, patients are followed up for 4 to 9 weeks.