Brief Summary: This phase IIa trial studies how well rintatolimod and pembrolizumab works in treating patients with colorectal cancer that does not respond to treatment (refractory), has spread to other places in the body (metastatic), or otherwise cannot be removed by surgery (unresectable). Rintatolimod is an immuno-oncology agent that can stimulate the immune system. Immunotherapy with monoclonal antibodies, such as pembrolizumab, may help the body's immune system attack the cancer, and may interfere with the ability of tumor cells to grow and spread. Giving rintatolimod and pembrolizumab together may work better than standard of care in treating patients with colorectal cancer.

Detailed Description: PRIMARY OBJECTIVE: I. Determine the objective response rate of patients with metastatic colorectal cancer (mCRC) treated with rintatolimod + pembrolizumab. SECONDARY OBJECTIVES: I. Establish the adverse event profile of combining rintatolimod and pembrolizumab. II. Estimate the median progression free survival and overall survival of patients with mCRC treated with rintatolimod and pembrolizumab. III. Determine the immune objective response rate of patients with mCRC treated with rintatolimod + pembrolizumab. EXPLORATORY OBJECTIVES: I. Assess modulation of the levels of CD8alpha expression and cytotoxic T-lymphocyte (CTL) density pre- and post-therapy. II. Assess chemokine levels in the tumor microenvironment and peripheral blood, including effector T cell (Teff)-attracting and regulatory T cell (Treg)-favoring chemokines. III. Characterize the fecal microbiotic profile and correlate those results with antitumor immune responses. OUTLINE: Patients receive rintatolimod intravenously (IV) over 30 minutes on days 1-3 and pembrolizumab IV over 30 minutes on day 3. Treatment repeats every 21 days for up to 3 cycles in the absence of disease progression or unacceptable toxicity. Beginning in cycle 4, patients receive rintatolimod IV over 30 minutes and pembrolizumab IV over 30 minutes on day 1. Cycles repeat every 21 days for up to 24 months from the first dose in the absence of disease progression or unacceptable toxicity. After completion of study treatment, patients are followed up at 30 and 90 days, and every 6 months for up to 2 years.