Brief Summary: This phase I trial investigates how well pembrolizumab and chemoradiotherapy works in treating patients with gastroesophageal cancer that cannot be removed by surgery (unresectable). Immunotherapy with monoclonal antibodies, such as pembrolizumab, may help the body's immune system attack the cancer, and may interfere with the ability of tumor cells to grow and spread. Chemotherapy drugs, such as fluorouracil, oxaliplatin and docetaxel work in different ways to stop the growth of tumor cells, either by killing the cells, by stopping them from dividing, or by stopping them from spreading. Radiation therapy uses high energy photons to kill tumor cells and shrink tumors. Giving pembrolizumab together with chemoradiotherapy may help to control gastroesophageal cancer.

Detailed Description: PRIMARY OBJECTIVE: I. To evaluate the clinical activity (as assessed by complete clinical response rate) of pembrolizumab plus chemoradiation. SECONDARY OBJECTIVES: I. To evaluate the safety and tolerability of pembrolizumab plus chemoradiation. II. To determine the overall survival efficacy of pembrolizumab plus chemoradiation. III. Evaluate progression free survival (PFS) by local investigator review and by blinded central radiologists' review. EXPLORATORY OBJECTIVES: I. To explore the association between PD-L1 expression by immunohistochemistry, somatic gene expression profiling, and clinical efficacy of pembrolizumab. II. To explore the relationship between genomic variation and response to study treatment. III. To identify molecular (genomic, metabolic, and/or proteomic) biomarkers that may correlate with clinical response/resistance, safety, pharmacodynamic activity, and/or the mechanism of action of pembrolizumab in combination with chemoradiation. IV. To determine the effect of pembrolizumab plus chemoradiation treatment on tumor T cell infiltration. OUTLINE: INDUCTION CHEMOTHERAPY: Patients receive pembrolizumab intravenously (IV) over 30 minutes on day 1. Treatment with pembrolizumab repeats every 3 weeks for up to 3 cycles in the absence of disease progression or unacceptable toxicity. Patients also receive oxaliplatin IV over 2 hours on day 1 followed by fluorouracil IV over 48 hours once every 2 weeks (Q2W) for 8 weeks (4 cycles) in the absence of disease progression or unacceptable toxicity. CONSOLIDATION CHEMORADIATION: After a 3 week treatment free period, patients receive pembrolizumab IV over 30 minutes on day 1. Treatment with pembrolizumab repeats every 3 weeks for up to 2 cycles in the absence of disease progression or unacceptable toxicity. Patients also receive fluorouracil IV continuous over 5 days (Monday through Friday) for 5 weeks and docetaxel IV over 1 hour once a week (QW) for 5 weeks in the absence of disease progression or unacceptable toxicity. Starting no later than 3 days after the beginning of the Consolidation Chemoradiation period, patients also undergo 28 fractions of radiation therapy in the absence of disease progression or unacceptable toxicity. After a 6-9 week treatment free period, patients continue pembrolizumab IV over 30 minutes on day 1. Treatment with pembrolizumab repeats every 3 weeks for up to 30 cycles in the absence of disease progression or unacceptable toxicity. After completion of study treatment, patients are followed up at 30 days, then every 9-12 weeks thereafter.